[jira] [Commented] (HBASE-6015) HBCK rerun should check all the regions which it checked in the first run

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277622#comment-13277622
 ] 

Anoop Sam John commented on HBASE-6015:
---

My doubt was like below
One region found to be problematic and it is fixed. In the rerun we should 
recheck this region right. Now the timelag factor can apply and can remove this 
region from checking in the rerun [If this region fix has changed the modtime 
of the HRegionInfo]

Also at the 1st run some of the regions might have been excluded because those 
are changed recently [As per the timelag value]  In the rerun do we need to 
check those regions? What if those at this time check happens and reports some 
issues?  Will it look like for the use that the HBCK was not able to fix the 
issues which it reported in the 1st run?

These are my thought after seeing the timelag concept... Pls correct me if my 
understanding is wrong  :)

> HBCK rerun should check all the regions which it checked in the first run
> -
>
> Key: HBASE-6015
> URL: https://issues.apache.org/jira/browse/HBASE-6015
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Reporter: Anoop Sam John
>
> When after the 1st round of run and possible fixes, HBCK does a rerun to 
> check the consistency of the regions. At this rerun
> 1.It should check all the regions which it checked in the 1st round. 
> 2.It should check only those regions which it checked in the 1st round. Might 
> be some other regions can come out of the timelag check at rerun time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277620#comment-13277620
 ] 

Hadoop QA commented on HBASE-5453:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527785/5453v11.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 27 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 32 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.client.TestMultipleTimestamps
  org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD
  
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
  org.apache.hadoop.hbase.replication.TestReplication
  org.apache.hadoop.hbase.client.TestTimestampsFilter
  org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
  org.apache.hadoop.hbase.client.TestFromClientSide
  org.apache.hadoop.hbase.mapreduce.TestImportExport
  org.apache.hadoop.hbase.replication.TestMasterReplication

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1906//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1906//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1906//console

This message is automatically generated.

> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> --
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 
> 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277619#comment-13277619
 ] 

Hudson commented on HBASE-5935:
---

Integrated in HBase-TRUNK #2889 (See 
[https://builds.apache.org/job/HBase-TRUNK/2889/])
HBASE-5935 Add Region-level PB-based calls to HMasterInterface (Revision 
1339488)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java
* /hbase/trunk/src/main/protobuf/RegionServerStatus.proto
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java


> Add Region-level PB-based calls to HMasterInterface
> ---
>
> Key: HBASE-5935
> URL: https://issues.apache.org/jira/browse/HBASE-5935
> Project: HBase
>  Issue Type: Task
>  Components: ipc, master, migration
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 0.96.0
>
> Attachments: HBASE-5935-v2.patch, HBASE-5935-v2.patch, 
> HBASE-5935-v3.patch
>
>
> This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
> also make this a subtask (apparently).
> This is for converting the region-level calls, i.e.:
> moveRegion
> assignRegion
> unassignRegion

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5840) Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing the old status

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277613#comment-13277613
 ] 

ramkrishna.s.vasudevan commented on HBASE-5840:
---

I will commit this to 0.94.1 today.  

> Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing 
> the old status
> --
>
> Key: HBASE-5840
> URL: https://issues.apache.org/jira/browse/HBASE-5840
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: rajeshbabu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: HBASE-5840.patch, HBASE-5840_trunk.patch, 
> HBASE-5840_v2.patch
>
>
> TaskMonitor Status will not be cleared in case Regions FAILED_OPEN. This will 
> keeps showing old status.
> This will miss leads the user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6015) HBCK rerun should check all the regions which it checked in the first run

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277611#comment-13277611
 ] 

Jonathan Hsieh commented on HBASE-6015:
---

Anoop -- I'm not sure I understand.  Is the idea that it *only* checks problem 
regions from a previous round?  Current behavior reloads all regions on the 
follow up rounds. 

> HBCK rerun should check all the regions which it checked in the first run
> -
>
> Key: HBASE-6015
> URL: https://issues.apache.org/jira/browse/HBASE-6015
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Reporter: Anoop Sam John
>
> When after the 1st round of run and possible fixes, HBCK does a rerun to 
> check the consistency of the regions. At this rerun
> 1.It should check all the regions which it checked in the 1st round. 
> 2.It should check only those regions which it checked in the 1st round. Might 
> be some other regions can come out of the timelag check at rerun time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6001) Upgrade slf4j to 1.6.1

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277608#comment-13277608
 ] 

Andrew Purtell commented on HBASE-6001:
---

I'll take care of this tomorrow morning if you haven't gotten to it first. 


> Upgrade slf4j to 1.6.1
> --
>
> Key: HBASE-6001
> URL: https://issues.apache.org/jira/browse/HBASE-6001
> Project: HBase
>  Issue Type: Task
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: hbase-6001.patch
>
>
> We need to upgrade slf4j to 1.6.1 since other hadoop components use 1.6.1 now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6013) Polish sharp edges from CopyTable

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277602#comment-13277602
 ] 

Jonathan Hsieh commented on HBASE-6013:
---

I will commit tomorrow to all affected versions unless I hear otherwise.

> Polish sharp edges from CopyTable
> -
>
> Key: HBASE-6013
> URL: https://issues.apache.org/jira/browse/HBASE-6013
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6, 0.92.1, 0.94.0, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-6013.patch
>
>
> CopyTable doesn't report errors when invalid arguments are specified.  For 
> example, having a typo in --peer.adr (such as --peer.addr or -peer.adr) 
> silently uses the default cluster and does a same-cluster copy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6001) Upgrade slf4j to 1.6.1

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277600#comment-13277600
 ] 

Jonathan Hsieh commented on HBASE-6001:
---

+1.  Andrew, do you want to commit or shall I? (I'd plan on doing 
0.92/0.94/trunk.)

> Upgrade slf4j to 1.6.1
> --
>
> Key: HBASE-6001
> URL: https://issues.apache.org/jira/browse/HBASE-6001
> Project: HBase
>  Issue Type: Task
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: hbase-6001.patch
>
>
> We need to upgrade slf4j to 1.6.1 since other hadoop components use 1.6.1 now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5945) Reduce buffer copies in IPC server response path

[Todd Lipcon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-5945:
---

Status: Patch Available  (was: Open)

> Reduce buffer copies in IPC server response path
> 
>
> Key: HBASE-5945
> URL: https://issues.apache.org/jira/browse/HBASE-5945
> Project: HBase
>  Issue Type: Improvement
>  Components: ipc
>Affects Versions: 0.96.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Attachments: buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt
>
>
> The new PB code is sloppy with buffers and makes several needless copies. 
> This increases GC time a lot. A few simple changes can cut this back down.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277599#comment-13277599
 ] 

Jonathan Hsieh commented on HBASE-6018:
---

I will commit if this passes the hadoopqabot.

> hbck fails with a RejectedExecutionException
> 
>
> Key: HBASE-6018
> URL: https://issues.apache.org/jira/browse/HBASE-6018
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.94.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-6018-v2.patch, hbase-6018.patch
>
>
> On a long running job 0.94.0rc3 cluster, we get to a point where hbck 
> consistently encounters this error and fails:
> {code}
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
>   at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
>   at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>   at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>   at 
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
>   at 
> org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
>   at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>   at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6018) hbck fails with a RejectedExecutionException

[Jonathan Hsieh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-6018:
--

Attachment: hbase-6018-v2.patch

Added a simple unit test that fails before the patch.

> hbck fails with a RejectedExecutionException
> 
>
> Key: HBASE-6018
> URL: https://issues.apache.org/jira/browse/HBASE-6018
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.94.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-6018-v2.patch, hbase-6018.patch
>
>
> On a long running job 0.94.0rc3 cluster, we get to a point where hbck 
> consistently encounters this error and fails:
> {code}
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
>   at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
>   at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>   at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>   at 
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
>   at 
> org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
>   at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>   at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6000) Cleanup where we keep .proto files

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-6000:
-

Attachment: 6000.txt

Retry.  Failure looks unrelated.

> Cleanup where we keep .proto files
> --
>
> Key: HBASE-6000
> URL: https://issues.apache.org/jira/browse/HBASE-6000
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: stack
>Assignee: stack
> Attachments: 6000.txt, 6000.txt
>
>
> I see Andrew for his pb work over in rest has .protos files under 
> src/main/resources.  We should unify where these files live.  The recently 
> added .protos place them under src/main/protobuf  Its confusing.
> The thift idl files are here under resources too.
> Seems like we should move src/main/protobuf under src/resources to be 
> consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6000) Cleanup where we keep .proto files

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-6000:
-

Status: Open  (was: Patch Available)

> Cleanup where we keep .proto files
> --
>
> Key: HBASE-6000
> URL: https://issues.apache.org/jira/browse/HBASE-6000
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: stack
>Assignee: stack
> Attachments: 6000.txt
>
>
> I see Andrew for his pb work over in rest has .protos files under 
> src/main/resources.  We should unify where these files live.  The recently 
> added .protos place them under src/main/protobuf  Its confusing.
> The thift idl files are here under resources too.
> Seems like we should move src/main/protobuf under src/resources to be 
> consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277596#comment-13277596
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
---

(Updated 2012-05-17 05:58:11.354772)


Review request for hbase.


Changes
---

Address Gregory's two review comments.


Summary
---

A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
  New  class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
  Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
  Hide the Reference#Range enums.  Don't let them out of this class.
  Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  ClusterId under zk has been renamed ZKClusterId.
  Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
  Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  Move the RegionInfo convertion up into HRegionInfo instead of here.
  Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  Range in Reference is no longer public.
  Range in Reference is no longer public.
M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
  ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
  Use new serialization utlity in HTD.
M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
  Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
  Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
  Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
  Rename
A b/src/main/protobuf/ClusterId.proto
  Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
  Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
  Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
  Added TableSchema and ColumnFamilySchema


This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15 
  src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3 
  src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af 
  src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059 
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
9e4ada9 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab 
  src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java 
PRE-CREATION 
  src/main/java/

[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5453:
-

Fix Version/s: 0.96.0
 Release Note: Convert Reference, .tableinfo, .regioninfo, hbase.version, 
and hbase.id files to have pb content.
 Hadoop Flags: Reviewed
   Status: Patch Available  (was: Open)

> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> --
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 
> 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5453:
-

Attachment: 5453v11.txt
5453v11.txt

v11 addressing Gregory's last review comments.

> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> --
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 
> 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277593#comment-13277593
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--



bq.  On 2012-05-17 05:41:53, Gregory Chanan wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 275
bq.  > 
bq.  >
bq.  > This is still here?  Or just reviewboard showing it for some reason?

Flotsam.  An abandoned direction that I failed to clean up.  Thanks.


bq.  On 2012-05-17 05:41:53, Gregory Chanan wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/FSUtils.java, line 91
bq.  > 
bq.  >
bq.  > This is what you are using to ensure the sizes cannot be the same?  
Very nice!
bq.  > 
bq.  > I cannot find a call-site for this, though, maybe I missed it?

Ditto. I was going to use this to write .regioninfo and .tableinfo but then I 
ran into your review.  Study this method which I'll purge in the next version.  
See how it puts serialized class at top, two '\n's, and then a toString on the 
class?  Thats how .regioninfo files and .tableinfo files are currently written. 
 My patch now changes it so these files only contained the serialized object... 
no '\n' and no toString.  I do this so its very unlikely a Writable length will 
class w/ the length of a pb'd content (See HRegion for where we write 
.regioninfo and in FSTableDescriptors for where we write .tableinfo only we 
don't need the 'trick' here since we have actually read the file and can see if 
its pb'd).

Thanks for the review G.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7951
---


On 2012-05-16 23:56:18, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 23:56:18)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.Range in Reference is no longer public.
bq.Range in Reference is no longer public.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq.ClusterId got renamed ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq.  

[jira] [Commented] (HBASE-6000) Cleanup where we keep .proto files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277591#comment-13277591
 ] 

Hadoop QA commented on HBASE-6000:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527779/6000.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 32 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.replication.TestMasterReplication

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1905//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1905//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1905//console

This message is automatically generated.

> Cleanup where we keep .proto files
> --
>
> Key: HBASE-6000
> URL: https://issues.apache.org/jira/browse/HBASE-6000
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: stack
>Assignee: stack
> Attachments: 6000.txt
>
>
> I see Andrew for his pb work over in rest has .protos files under 
> src/main/resources.  We should unify where these files live.  The recently 
> added .protos place them under src/main/protobuf  Its confusing.
> The thift idl files are here under resources too.
> Seems like we should move src/main/protobuf under src/resources to be 
> consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277589#comment-13277589
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7951
---



src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java


This is still here?  Or just reviewboard showing it for some reason?



src/main/java/org/apache/hadoop/hbase/util/FSUtils.java


This is what you are using to ensure the sizes cannot be the same?  Very 
nice!

I cannot find a call-site for this, though, maybe I missed it?


- Gregory


On 2012-05-16 23:56:18, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 23:56:18)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.Range in Reference is no longer public.
bq.Range in Reference is no longer public.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq.ClusterId got renamed ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq.Use new serialization utlity in HTD.
bq.  M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.Generic method for writing dot file content.
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Reference#Range is not public any more
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq.Deprecated getHRegionInfo, etc.
bq.  D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq.  A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq.Rename
bq.  A b/src/main/protobuf/ClusterId.proto
bq.Added file for ClusterId only since its written to fs and to zk.
bq.  A b/src/main/protobuf/FS.proto
bq.Protos for fs files.
bq.  M b/src/main/protobuf/ZooKeeper.proto
bq.Moved ClusterId out to own proto file
bq.  M b/src/main/protobuf/hbase.proto
bq.Added TableSchema and ColumnFamilySchema
bq.  
bq.  
bq.  This addresses bug hbase-5453.
bq.  https://issues.apache.org/jira/browse/hbase-5453
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/jav

[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277584#comment-13277584
 ] 

stack commented on HBASE-6028:
--

IIRC, there already is a cancel facility that is used on region close; if an 
ongoing compaction, its interrupted.  If such is the case, this issue would 
hopefully just be about exposing this functionality in hbase admin (Sorry D, 
should have remembered today that we do this).

> Implement a cancel for in-progress compactions
> --
>
> Key: HBASE-6028
> URL: https://issues.apache.org/jira/browse/HBASE-6028
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Derek Wollenstein
>Priority: Minor
>  Labels: compaction, operations, regionserver
>
> Depending on current server load, it can be extremely expensive to run 
> periodic minor / major compactions.  It would be helpful to have a feature 
> where a user could use the shell or a client tool to explicitly cancel an 
> in-progress compactions.  This would allow a system to recover when too many 
> regions became eligible for compactions at once

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277581#comment-13277581
 ] 

Jonathan Hsieh commented on HBASE-6018:
---

I'm convinced that SynchronousQueue was just wrong and the failure was 
"expected behavior".  I'm a little confused by your statement Lars; queues are 
just used if there are more tasks than max concurrent enough threads in the 
pool.

See java doc here 
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/ThreadPoolExecutor.html

{quote}
Queuing
Any BlockingQueue may be used to transfer and hold submitted tasks. The use of 
this queue interacts with pool sizing:
If fewer than corePoolSize threads are running, the Executor always prefers 
adding a new thread rather than queuing.
If corePoolSize or more threads are running, the Executor always prefers 
queuing a request rather than adding a new thread.
If a request cannot be queued, a new thread is created unless this would exceed 
maximumPoolSize, in which case, the task will be rejected.
{quote}

> hbck fails with a RejectedExecutionException
> 
>
> Key: HBASE-6018
> URL: https://issues.apache.org/jira/browse/HBASE-6018
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.94.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-6018.patch
>
>
> On a long running job 0.94.0rc3 cluster, we get to a point where hbck 
> consistently encounters this error and fails:
> {code}
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
>   at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
>   at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>   at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>   at 
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
>   at 
> org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
>   at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>   at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6029) HBCK doesn't recover Balance switch if exception occurs in onlineHbck().

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277582#comment-13277582
 ] 

stack commented on HBASE-6029:
--

+1

> HBCK doesn't recover Balance switch if exception occurs in onlineHbck().
> 
>
> Key: HBASE-6029
> URL: https://issues.apache.org/jira/browse/HBASE-6029
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 0.94.0
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Fix For: 0.96.0, 0.94.1
>
> Attachments: HBASE-6029.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5935:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to TRUNK.  Thanks for the patch Gregory.

> Add Region-level PB-based calls to HMasterInterface
> ---
>
> Key: HBASE-5935
> URL: https://issues.apache.org/jira/browse/HBASE-5935
> Project: HBase
>  Issue Type: Task
>  Components: ipc, master, migration
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 0.96.0
>
> Attachments: HBASE-5935-v2.patch, HBASE-5935-v2.patch, 
> HBASE-5935-v3.patch
>
>
> This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
> also make this a subtask (apparently).
> This is for converting the region-level calls, i.e.:
> moveRegion
> assignRegion
> unassignRegion

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6029) HBCK doesn't recover Balance switch if exception occurs in onlineHbck().

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277560#comment-13277560
 ] 

Hadoop QA commented on HBASE-6029:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527766/HBASE-6029.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 31 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1904//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1904//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1904//console

This message is automatically generated.

> HBCK doesn't recover Balance switch if exception occurs in onlineHbck().
> 
>
> Key: HBASE-6029
> URL: https://issues.apache.org/jira/browse/HBASE-6029
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 0.94.0
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Fix For: 0.96.0, 0.94.1
>
> Attachments: HBASE-6029.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6000) Cleanup where we keep .proto files

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-6000:
-

 Assignee: stack
Affects Version/s: 0.96.0
 Release Note: Moves the REST proto files from src/main/resource to 
src/main/protobuf
   Status: Patch Available  (was: Open)

> Cleanup where we keep .proto files
> --
>
> Key: HBASE-6000
> URL: https://issues.apache.org/jira/browse/HBASE-6000
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: stack
>Assignee: stack
> Attachments: 6000.txt
>
>
> I see Andrew for his pb work over in rest has .protos files under 
> src/main/resources.  We should unify where these files live.  The recently 
> added .protos place them under src/main/protobuf  Its confusing.
> The thift idl files are here under resources too.
> Seems like we should move src/main/protobuf under src/resources to be 
> consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6000) Cleanup where we keep .proto files

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-6000:
-

Attachment: 6000.txt

Here's a patch that moves it all under src/main/protobuf

> Cleanup where we keep .proto files
> --
>
> Key: HBASE-6000
> URL: https://issues.apache.org/jira/browse/HBASE-6000
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
> Attachments: 6000.txt
>
>
> I see Andrew for his pb work over in rest has .protos files under 
> src/main/resources.  We should unify where these files live.  The recently 
> added .protos place them under src/main/protobuf  Its confusing.
> The thift idl files are here under resources too.
> Seems like we should move src/main/protobuf under src/resources to be 
> consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5923) Cleanup checkAndXXX logic

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277552#comment-13277552
 ] 

Lars Hofhansl commented on HBASE-5923:
--

Looking at this again.

@Stack: How would you abstract the PB dependency in HTableInterface away in 
trunk?


> Cleanup checkAndXXX logic
> -
>
> Key: HBASE-5923
> URL: https://issues.apache.org/jira/browse/HBASE-5923
> Project: HBase
>  Issue Type: Improvement
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5923-0.94.txt, 5923-trunk.txt
>
>
> 1. the checkAnd{Put|Delete} method that takes a CompareOP is not exposed via 
> HTable[Interface].
> 2. there is unnecessary duplicate code in the check{Put|Delete} code in 
> HRegionServer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6000) Cleanup where we keep .proto files

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277554#comment-13277554
 ] 

stack commented on HBASE-6000:
--

There is no requirement that you install protoc to build hbase; the generated 
classes are checked in as they are for thrift and avro.  Generated stuff is at 
src/main/java/org/apache/hadoop/hbase/protobuf/generated.

> Cleanup where we keep .proto files
> --
>
> Key: HBASE-6000
> URL: https://issues.apache.org/jira/browse/HBASE-6000
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>
> I see Andrew for his pb work over in rest has .protos files under 
> src/main/resources.  We should unify where these files live.  The recently 
> added .protos place them under src/main/protobuf  Its confusing.
> The thift idl files are here under resources too.
> Seems like we should move src/main/protobuf under src/resources to be 
> consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5546) Master assigns region in the original region server when opening region failed

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277551#comment-13277551
 ] 

ramkrishna.s.vasudevan commented on HBASE-5546:
---

Will commit today unless objections.

> Master assigns region in the original region server when opening region 
> failed  
> 
>
> Key: HBASE-5546
> URL: https://issues.apache.org/jira/browse/HBASE-5546
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: gaojinchao
>Assignee: Ashutosh Jindal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: hbase-5546.patch, hbase-5546_1.patch, 
> hbase-5546_2.patch, hbase-5546_3.patch
>
>
> Master assigns region in the original region server when 
> RS_ZK_REGION_FAILED_OPEN envent was coming.
> Maybe we should choose other region server.
> [2012-03-07 10:14:21,750] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:31,826] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:41,903] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:51,975] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:02,056] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:12,167] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:22,231] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:32,303] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:42,375] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:52,447] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:02,528] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:12,600] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:22,676] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5882) Prcoess RIT on master restart can try assigning the region if the region is found on a dead server instead of waiting for Timeout Monitor

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277550#comment-13277550
 ] 

ramkrishna.s.vasudevan commented on HBASE-5882:
---

Will commit today unless objections.  Will make the method name as 
'wasOnDeadServer'. Any comments pls share.

> Prcoess RIT on master restart can try assigning the region if the region is 
> found on a dead server instead of waiting for Timeout Monitor
> -
>
> Key: HBASE-5882
> URL: https://issues.apache.org/jira/browse/HBASE-5882
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.6, 0.92.1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: hbase_5882.patch
>
>
> Currently on  master restart if it tries to do processRIT, any region if 
> found on dead server tries to avoid the nwe assignment so that timeout 
> monitor can take care.
> This case is more prominent if the node is found in RS_ZK_REGION_OPENING 
> state. I think we can handle this by triggering a new assignment with a new 
> plan.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6025) Expose Hadoop Dynamic Metrics through JSON Rest interface

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277545#comment-13277545
 ] 

stack commented on HBASE-6025:
--

@Enis You saying that what comes out of jmx will show in a hadoop webapp 
servlet mounted at /jmx?  I did not know such a thing existed (I forgot 
hbase-5309).

I tried it.  Had to copy in jackson and newer hadoop (we ship 1.0.0 even in 
0.94 which needs to be fixed).  I see /jmx metrics (We should add a 'metrics' 
link along the top beside the log level, thread dump, etc. servlets.

What other servlets are we missing out on @Enis?  There was a configuration 
dump IIRC but maybe thats not till 2.0.0 hadoop?

Oh, it looks like the jmx per-region stuff is showing under /jmx because 
Elliott already added this to 0.94 and trunk.

So, seems like, to make this work for trunk, all we have to do is:

1. update jackson and hadoop versions
2. Add a metrics link to show on all pages
3. Fix this exception:

{code}
2012-05-16 21:24:06,077 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting 
attribute CollectionUsageThresholdCount of java.lang:type=MemoryPool,name=Code 
Cache threw an exception
javax.management.RuntimeMBeanException: 
java.lang.UnsupportedOperationException: CollectionUsage threshold is not 
supported
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at 
org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:251)
at 
org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:227)
at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:159)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.UnsupportedOperationException: CollectionUsage threshold 
is not supported
at 
sun.management.MemoryPoolImpl.getCollectionUsageThresholdCount(MemoryPoolImpl.java:229)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
{code}

Should at least be backported to 0.94.

This is great.

> Expose Hadoop Dynamic Metrics through JSON Rest interface
> -
>
> Key: HBASE-6025
> URL: https://issues.apache.org/jira/browse/HBASE-6025
> Project: HBase
>  Issue Type: Improvement
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http

[jira] [Updated] (HBASE-5987) HFileBlockIndex improvement

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5987:
---

Attachment: D3237.8.patch

Liyin updated the revision "[jira][89-fb] [HBASE-5987] HFileBlockIndex 
improvement".
Reviewers: Kannan, mbautin

  A wrong diff is attached here previous. Sorry for the spam.

REVISION DETAIL
  https://reviews.facebook.net/D3237

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/HConstants.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java

To: Kannan, mbautin, Liyin
Cc: JIRA, todd, tedyu


> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, 
> D3237.4.patch, D3237.5.patch, D3237.6.patch, D3237.7.patch, D3237.8.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value for each data block and the target 
> key value is beween the last key value of data block N-1 and data block N, 
> then the target key value is supposed be data block N for sure. 
> As long as HBase only supports the forward scan, the last key value makes 
> more sense to be indexed on than the start key value. 
> Thanks Kannan and Mikhail for the insightful discussions and suggestions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5776) HTableMultiplexer

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5776:
---

Attachment: D2775.3.patch

Liyin updated the revision "[jira][89-fb][HBASE-5776] HTableMultiplexer".
Reviewers: Kannan

  Address Kannan's comments.

REVISION DETAIL
  https://reviews.facebook.net/D2775

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/HConstants.java
  src/main/java/org/apache/hadoop/hbase/client/HConnection.java
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  src/main/java/org/apache/hadoop/hbase/client/HTable.java
  src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
  src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
  src/main/java/org/apache/hadoop/hbase/client/MultiplexablePut.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java

To: Kannan, Liyin
Cc: JIRA, tedyu


> HTableMultiplexer 
> --
>
> Key: HBASE-5776
> URL: https://issues.apache.org/jira/browse/HBASE-5776
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, 
> D2775.2.patch, D2775.3.patch
>
>
> There is a known issue in HBase client that single slow/dead region server 
> could slow down the multiput operations across all the region servers. So the 
> HBase client will be as slow as the slowest region server in the cluster. 
>  
> To solve this problem, HTableMultiplexer will separate the multiput 
> submitting threads with the flush threads, which means the multiput operation 
> will be a nonblocking operation. 
> The submitting thread will shard all the puts into different queues based on 
> its destination region server and return immediately. The flush threads will 
> flush these puts from each queue to its destination region server. 
> Currently the HTableMultiplexer only supports the put operation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6029) HBCK doesn't recover Balance switch if exception occurs in onlineHbck().

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277517#comment-13277517
 ] 

Jimmy Xiang commented on HBASE-6029:


+1

> HBCK doesn't recover Balance switch if exception occurs in onlineHbck().
> 
>
> Key: HBASE-6029
> URL: https://issues.apache.org/jira/browse/HBASE-6029
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 0.94.0
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Fix For: 0.96.0, 0.94.1
>
> Attachments: HBASE-6029.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277516#comment-13277516
 ] 

Hadoop QA commented on HBASE-5986:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527760/5986-v2.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 32 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1903//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1903//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1903//console

This message is automatically generated.

> Clients can see holes in the META table when regions are being split
> 
>
> Key: HBASE-5986
> URL: https://issues.apache.org/jira/browse/HBASE-5986
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch
>
>
> We found this issue when running large scale ingestion tests for HBASE-5754. 
> The problem is that the .META. table updates are not atomic while splitting a 
> region. In SplitTransaction, there is a time lap between the marking the 
> parent offline, and adding of daughters to the META table. This can result in 
> clients using MetaScanner, of HTable.getStartEndKeys (used by the 
> TableInputFormat) missing regions which are made just offline, but the 
> daughters are not added yet. 
> This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6029) HBCK doesn't recover Balance switch if exception occurs in onlineHbck().

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-6029:
--

Fix Version/s: 0.94.1
   0.96.0
 Assignee: Maryann Xue
 Hadoop Flags: Reviewed

Patch looks good.

> HBCK doesn't recover Balance switch if exception occurs in onlineHbck().
> 
>
> Key: HBASE-6029
> URL: https://issues.apache.org/jira/browse/HBASE-6029
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 0.94.0
>Reporter: Maryann Xue
>Assignee: Maryann Xue
> Fix For: 0.96.0, 0.94.1
>
> Attachments: HBASE-6029.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6030) Log AccessDeniedExceptions at INFO level

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-6030:
--

Component/s: coprocessors
   Assignee: Andrew Purtell

> Log AccessDeniedExceptions at INFO level
> 
>
> Key: HBASE-6030
> URL: https://issues.apache.org/jira/browse/HBASE-6030
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors, security
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
> Environment: AccessController coprocessor
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>
> The HBase AccessController can emit a detailed trace of every action, and 
> whether it succeeded or failed, but this is expected to be used only for 
> debugging an application in a staging environment. In production a 
> RegionServer may have thousands of requests per second, logging the audit 
> trace just isn't viable. However, we might want to log the 
> AccessDeniedExceptions which result from access failures in the daemon logs 
> like Hadoop does, e.g. the NameNode log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6029) HBCK doesn't recover Balance switch if exception occurs in onlineHbck().

[Maryann Xue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryann Xue updated HBASE-6029:
---

Attachment: HBASE-6029.patch

add try-finally block to recover balance switch.

> HBCK doesn't recover Balance switch if exception occurs in onlineHbck().
> 
>
> Key: HBASE-6029
> URL: https://issues.apache.org/jira/browse/HBASE-6029
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 0.94.0
>Reporter: Maryann Xue
> Attachments: HBASE-6029.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6029) HBCK doesn't recover Balance switch if exception occurs in onlineHbck().

[Maryann Xue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryann Xue updated HBASE-6029:
---

Status: Patch Available  (was: Open)

> HBCK doesn't recover Balance switch if exception occurs in onlineHbck().
> 
>
> Key: HBASE-6029
> URL: https://issues.apache.org/jira/browse/HBASE-6029
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 0.94.0
>Reporter: Maryann Xue
> Attachments: HBASE-6029.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6030) Log AccessDeniedExceptions at INFO level

[Andrew Purtell (JIRA)]

Andrew Purtell created HBASE-6030:
-

 Summary: Log AccessDeniedExceptions at INFO level
 Key: HBASE-6030
 URL: https://issues.apache.org/jira/browse/HBASE-6030
 Project: HBase
  Issue Type: Improvement
  Components: security
Affects Versions: 0.92.2, 0.96.0, 0.94.1
 Environment: AccessController coprocessor
Reporter: Andrew Purtell
Priority: Minor


The HBase AccessController can emit a detailed trace of every action, and 
whether it succeeded or failed, but this is expected to be used only for 
debugging an application in a staging environment. In production a RegionServer 
may have thousands of requests per second, logging the audit trace just isn't 
viable. However, we might want to log the AccessDeniedExceptions which result 
from access failures in the daemon logs like Hadoop does, e.g. the NameNode log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6029) HBCK doesn't recover Balance switch if exception occurs in onlineHbck().

[Maryann Xue (JIRA)]

Maryann Xue created HBASE-6029:
--

 Summary: HBCK doesn't recover Balance switch if exception occurs 
in onlineHbck().
 Key: HBASE-6029
 URL: https://issues.apache.org/jira/browse/HBASE-6029
 Project: HBase
  Issue Type: Bug
  Components: hbck
Reporter: Maryann Xue




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6029) HBCK doesn't recover Balance switch if exception occurs in onlineHbck().

[Maryann Xue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maryann Xue updated HBASE-6029:
---

Affects Version/s: 0.94.0

> HBCK doesn't recover Balance switch if exception occurs in onlineHbck().
> 
>
> Key: HBASE-6029
> URL: https://issues.apache.org/jira/browse/HBASE-6029
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 0.94.0
>Reporter: Maryann Xue
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277494#comment-13277494
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7948
---


v2 looks much better.


src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java


We may need to mark this JIRA an incompatible change.



src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


'with' -> 'by'



src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


This class can be private.



src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


The interval can be made shorter.



src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


Name this method verifyRegionsOfHTable ?


- Ted


On 2012-05-17 00:58:21, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-17 00:58:21)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this issue when running large scale ingestion tests for 
HBASE-5754. The problem is that the .META. table updates are not atomic while 
splitting a region. In SplitTransaction, there is a time lap between the 
marking the parent offline, and adding of daughters to the META table. This can 
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the 
TableInputFormat) missing regions which are made just offline, but the 
daughters are not added yet.
bq.  
bq.  This patch is the approach 2 mentioned in the issue comments, mainly 
during META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 
bq.  
bq.  Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 
bq.  
bq.  
bq.  This addresses bug HBASE-5986.
bq.  https://issues.apache.org/jira/browse/HBASE-5986
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 
5cac9af 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 
bq.  
bq.  Diff: https://reviews.apache.org/r/5133/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  added extensive tests under TestEndToEndSplitTranscation, and ran existing 
unit tests. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  enis
bq.  
bq.



> Clients can see holes in the META table when regions are being split
> 
>
> Key: HBASE-5986
> URL: https://issues.apache.org/jira/browse/HBASE-5986
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch
>
>
> We found this issue when running large scale ingestion tests for HBASE-5754. 
> The problem is that the .META. table updates are not atomic while splitting a 
> region. In SplitTransaction, there is a time lap between the marking the 
> parent offline, and adding of daughters to the META table. This can result in 
> clients using MetaScanner, of HTable.getStartEndKeys (used by the 
> TableInputFormat) missing regions which are made just offline, but the 
> daughters are not added yet. 
> This is also related to HBASE-4335. 

--
This message is automatically generated

[jira] [Updated] (HBASE-5986) Clients can see holes in the META table when regions are being split

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5986:
--

Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

> Clients can see holes in the META table when regions are being split
> 
>
> Key: HBASE-5986
> URL: https://issues.apache.org/jira/browse/HBASE-5986
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch
>
>
> We found this issue when running large scale ingestion tests for HBASE-5754. 
> The problem is that the .META. table updates are not atomic while splitting a 
> region. In SplitTransaction, there is a time lap between the marking the 
> parent offline, and adding of daughters to the META table. This can result in 
> clients using MetaScanner, of HTable.getStartEndKeys (used by the 
> TableInputFormat) missing regions which are made just offline, but the 
> daughters are not added yet. 
> This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5986) Clients can see holes in the META table when regions are being split

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5986:
--

Attachment: 5986-v2.txt

v2 from Enis

> Clients can see holes in the META table when regions are being split
> 
>
> Key: HBASE-5986
> URL: https://issues.apache.org/jira/browse/HBASE-5986
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch
>
>
> We found this issue when running large scale ingestion tests for HBASE-5754. 
> The problem is that the .META. table updates are not atomic while splitting a 
> region. In SplitTransaction, there is a time lap between the marking the 
> parent offline, and adding of daughters to the META table. This can result in 
> clients using MetaScanner, of HTable.getStartEndKeys (used by the 
> TableInputFormat) missing regions which are made just offline, but the 
> daughters are not added yet. 
> This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6028) Implement a cancel for in-progress compactions

[Derek Wollenstein (JIRA)]

Derek Wollenstein created HBASE-6028:


 Summary: Implement a cancel for in-progress compactions
 Key: HBASE-6028
 URL: https://issues.apache.org/jira/browse/HBASE-6028
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Reporter: Derek Wollenstein
Priority: Minor


Depending on current server load, it can be extremely expensive to run periodic 
minor / major compactions.  It would be helpful to have a feature where a user 
could use the shell or a client tool to explicitly cancel an in-progress 
compactions.  This would allow a system to recover when too many regions became 
eligible for compactions at once

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5994) Document and add to TableMapReduceUtil setup if security is enabled

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-5994:
--

Description: 
If secure RPC is enabled, then users must enable the TokenProvider coprocessor. 
They also must acquire a job token and add it to the job configuration, by way 
of something like:

{code}
if (User.isHBaseSecurityEnabled()) try {
  User.getCurrent().obtainAuthTokenForJob(conf, job);
} catch (InterruptedException ie) {
  Thread.interrupted();
}
{code}

TableMapReduceUtil will do this automatically.

Add some detail about this to the manual.

  was:
If secure RPC is enabled, then users must enable the TokenProvider coprocessor. 
They also must acquire a job token and add it to the job configuration, by way 
of something like:

{code}
if (User.isSecurityEnabled()) try {
  User.getCurrent().obtainAuthTokenForJob(conf, job);
} catch (InterruptedException ie) {
  Thread.interrupted();
}
{code}

Add this to the manual and update TableMapReduceUtil.


> Document and add to TableMapReduceUtil setup if security is enabled
> ---
>
> Key: HBASE-5994
> URL: https://issues.apache.org/jira/browse/HBASE-5994
> Project: HBase
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>
> If secure RPC is enabled, then users must enable the TokenProvider 
> coprocessor. They also must acquire a job token and add it to the job 
> configuration, by way of something like:
> {code}
> if (User.isHBaseSecurityEnabled()) try {
>   User.getCurrent().obtainAuthTokenForJob(conf, job);
> } catch (InterruptedException ie) {
>   Thread.interrupted();
> }
> {code}
> TableMapReduceUtil will do this automatically.
> Add some detail about this to the manual.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277482#comment-13277482
 ] 

Phabricator commented on HBASE-5987:


tedyu has commented on the revision "[jira][89-fb] [HBASE-5987] HFileBlockIndex 
improvement".

  This patch shouldn't include HTableMultiplexer changes, right ?

REVISION DETAIL
  https://reviews.facebook.net/D3237

BRANCH
  HBASE-5776

To: Kannan, mbautin, Liyin
Cc: JIRA, todd, tedyu


> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, 
> D3237.4.patch, D3237.5.patch, D3237.6.patch, D3237.7.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value for each data block and the target 
> key value is beween the last key value of data block N-1 and data block N, 
> then the target key value is supposed be data block N for sure. 
> As long as HBase only supports the forward scan, the last key value makes 
> more sense to be indexed on than the start key value. 
> Thanks Kannan and Mikhail for the insightful discussions and suggestions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6025) Expose Hadoop Dynamic Metrics through JSON Rest interface

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277481#comment-13277481
 ] 

Enis Soztutar commented on HBASE-6025:
--

Makes sense. Thanks for changing the issue title. Is the plan to add them to 
jmx, which in turn will make them available under /jmx? 

> Expose Hadoop Dynamic Metrics through JSON Rest interface
> -
>
> Key: HBASE-6025
> URL: https://issues.apache.org/jira/browse/HBASE-6025
> Project: HBase
>  Issue Type: Improvement
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5987) HFileBlockIndex improvement

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5987:
---

Attachment: D3237.7.patch

Liyin updated the revision "[jira][89-fb] [HBASE-5987] HFileBlockIndex 
improvement".
Reviewers: Kannan, mbautin

  Address Kannan's comments.

REVISION DETAIL
  https://reviews.facebook.net/D3237

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/HConstants.java
  src/main/java/org/apache/hadoop/hbase/client/HConnection.java
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  src/main/java/org/apache/hadoop/hbase/client/HTable.java
  src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
  src/main/java/org/apache/hadoop/hbase/client/MultiPut.java
  src/main/java/org/apache/hadoop/hbase/client/MultiplexablePut.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java

To: Kannan, mbautin, Liyin
Cc: JIRA, todd, tedyu


> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, 
> D3237.4.patch, D3237.5.patch, D3237.6.patch, D3237.7.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value for each data block and the target 
> key value is beween the last key value of data block N-1 and data block N, 
> then the target key value is supposed be data block N for sure. 
> As long as HBase only supports the forward scan, the last key value makes 
> more sense to be indexed on than the start key value. 
> Thanks Kannan and Mikhail for the insightful discussions and suggestions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6027) Update the reference guide to reflect the changes in the security profile

[Devaraj Das (JIRA)]

Devaraj Das created HBASE-6027:
--

 Summary: Update the reference guide to reflect the changes in the 
security profile
 Key: HBASE-6027
 URL: https://issues.apache.org/jira/browse/HBASE-6027
 Project: HBase
  Issue Type: Bug
Reporter: Devaraj Das


The refguide needs to be updated to reflect the fact that there is no security 
profile anymore, etc. [Follow up to HBASE-5732]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277309#comment-13277309
 ] 

Phabricator commented on HBASE-5987:


tedyu has commented on the revision "[jira][89-fb] [HBASE-5987] HFileBlockIndex 
improvement".

INLINE COMMENTS
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:2 
No year, please.
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:41 
This class doesn't extend HBaseTestCase but uses methods from HBaseTestCase.

  It would be better to not reference HBaseTestCase.

REVISION DETAIL
  https://reviews.facebook.net/D3237

BRANCH
  HBASE-5987-fb

To: Kannan, mbautin, Liyin
Cc: JIRA, todd, tedyu


> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, 
> D3237.4.patch, D3237.5.patch, D3237.6.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value for each data block and the target 
> key value is beween the last key value of data block N-1 and data block N, 
> then the target key value is supposed be data block N for sure. 
> As long as HBase only supports the forward scan, the last key value makes 
> more sense to be indexed on than the start key value. 
> Thanks Kannan and Mikhail for the insightful discussions and suggestions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277306#comment-13277306
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/
---

(Updated 2012-05-17 00:58:21.965488)


Review request for hbase.


Changes
---

v2: 
 - addressed review comments
 - when timeout and interrupt throws RegionOfflineException. 
 - reuse HTable instances in BMSV. 
 - adds close() to MSV assuming it is indeed not public, evolving but private. 
we should decide the visibility. 


Summary
---

We found this issue when running large scale ingestion tests for HBASE-5754. 
The problem is that the .META. table updates are not atomic while splitting a 
region. In SplitTransaction, there is a time lap between the marking the parent 
offline, and adding of daughters to the META table. This can result in clients 
using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat) 
missing regions which are made just offline, but the daughters are not added 
yet.

This patch is the approach 2 mentioned in the issue comments, mainly during 
META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 

Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 


This addresses bug HBASE-5986.
https://issues.apache.org/jira/browse/HBASE-5986


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af 
  src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
  src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 
  src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 

Diff: https://reviews.apache.org/r/5133/diff


Testing
---

added extensive tests under TestEndToEndSplitTranscation, and ran existing unit 
tests. 


Thanks,

enis



> Clients can see holes in the META table when regions are being split
> 
>
> Key: HBASE-5986
> URL: https://issues.apache.org/jira/browse/HBASE-5986
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-5986-test_v1.patch
>
>
> We found this issue when running large scale ingestion tests for HBASE-5754. 
> The problem is that the .META. table updates are not atomic while splitting a 
> region. In SplitTransaction, there is a time lap between the marking the 
> parent offline, and adding of daughters to the META table. This can result in 
> clients using MetaScanner, of HTable.getStartEndKeys (used by the 
> TableInputFormat) missing regions which are made just offline, but the 
> daughters are not added yet. 
> This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6025) Expose Hadoop Dynamic Metrics through JSON Rest interface

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-6025:
-

Summary: Expose Hadoop Dynamic Metrics through JSON Rest interface  (was: 
Expose Hadoop Metrics through JSON Rest interface)

> Expose Hadoop Dynamic Metrics through JSON Rest interface
> -
>
> Key: HBASE-6025
> URL: https://issues.apache.org/jira/browse/HBASE-6025
> Project: HBase
>  Issue Type: Improvement
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6025) Expose Hadoop Metrics through JSON Rest interface

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277280#comment-13277280
 ] 

Elliott Clark commented on HBASE-6025:
--

RegionServerDynamicStatistics need to be exposed.  Sorry I wasn't very clear.

> Expose Hadoop Metrics through JSON Rest interface
> -
>
> Key: HBASE-6025
> URL: https://issues.apache.org/jira/browse/HBASE-6025
> Project: HBase
>  Issue Type: Improvement
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6025) Expose Hadoop Metrics through JSON Rest interface

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277279#comment-13277279
 ] 

Enis Soztutar commented on HBASE-6025:
--

How is this different than http://region-server:60030/jmx ? It works with 
hadoop-1.0.1+ (HBASE-5309)

> Expose Hadoop Metrics through JSON Rest interface
> -
>
> Key: HBASE-6025
> URL: https://issues.apache.org/jira/browse/HBASE-6025
> Project: HBase
>  Issue Type: Improvement
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6019) [refGuide] ported pseudo-distributed.html page to RefGuide Config chapter

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277278#comment-13277278
 ] 

stack commented on HBASE-6019:
--

Sweet. Thanks for updating the site.  I'll publish in a mo.

> [refGuide] ported pseudo-distributed.html page to RefGuide Config chapter
> -
>
> Key: HBASE-6019
> URL: https://issues.apache.org/jira/browse/HBASE-6019
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: hbase_hbase_6019.patch
>
>
> Ported the separate pseudo-distributed.html page that existed outside the 
> RefGuie into the Configuration chapter.
> configuration.xml
> * Added an example of a local pseudo-dist HDFS config file (the refguide 
> didn't have this)
> * Ported pseudo-dist extras to pseudo-dist section.
> pseudo-distributed.xml
> * This is the old page.  I'm leaving this backward compatibility so that the 
> old links don't break, although the only thing this says now is the content 
> has been moved to the RefGuide
> site.xml
> * Removing pseudo-dist extras from the left-hand nav.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6025) Expose Hadoop Metrics through JSON Rest interface

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277276#comment-13277276
 ] 

stack commented on HBASE-6025:
--

A few folks have asked for this including a gentleman w/ gallic accent (A 
'real' fr accent, not that quebecoispatois).

> Expose Hadoop Metrics through JSON Rest interface
> -
>
> Key: HBASE-6025
> URL: https://issues.apache.org/jira/browse/HBASE-6025
> Project: HBase
>  Issue Type: Improvement
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6025) Expose Hadoop Metrics through JSON Rest interface

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277275#comment-13277275
 ] 

stack commented on HBASE-6025:
--

One option is enable REST webapp and then the client can ask for the their 
metrics as pb, json, etc. (It does this encoding for you).

> Expose Hadoop Metrics through JSON Rest interface
> -
>
> Key: HBASE-6025
> URL: https://issues.apache.org/jira/browse/HBASE-6025
> Project: HBase
>  Issue Type: Improvement
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277272#comment-13277272
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--



bq.  On 2012-05-16 22:12:10, Gregory Chanan wrote:
bq.  > In a previous comment you said (about the 
HTableDescriptor/HColumnDesriptor pb stuff):
bq.  > "Well, Andrew beat us both to it over in his REST pb stuff.  We need to 
reconcile his w/ ours too"
bq.  > 
bq.  > Where is his stuff?  I couldn't find it.  Should we create a JIRA about 
reconciling? It would be nice to have something, however imperfect, up in trunk 
to work against, then we could fix up later.

Its under src/main/resources/org  I just tripped over it myself yesterday.  
 I made HBASE-6026 to do the reconcile (after HBASE-6000 goes in).

Are you down w/ the change in .regioninfo Gregory as means of 'ensuring' we 
don't have a Writable and pb serialization end up as same size?

Thanks for review.


bq.  On 2012-05-16 22:12:10, Gregory Chanan wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 276
bq.  > 

bq.  >
bq.  > This isn't ever read?

This is gone now.  I used another technique figuring if object serialized -- 
read file fully into byte array and test for the pb prefix -- rather than this 
hacky setting attribute on class.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7944
---


On 2012-05-16 17:02:35, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 17:02:35)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.Range in Reference is no longer public.
bq.Range in Reference is no longer public.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq.ClusterId got renamed ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq.Use new serialization utlity in HTD.
bq.  M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.Generic method for writing dot file content.
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Reference#Range is not public any more
bq.  M

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277273#comment-13277273
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
---

(Updated 2012-05-16 23:56:18.496225)


Review request for hbase.


Changes
---

v10... the last thing I added to JIRA


Summary
---

A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
  New  class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
  Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
  Hide the Reference#Range enums.  Don't let them out of this class.
  Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  ClusterId under zk has been renamed ZKClusterId.
  Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
  Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  Move the RegionInfo convertion up into HRegionInfo instead of here.
  Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  Range in Reference is no longer public.
  Range in Reference is no longer public.
M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
  ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
  Use new serialization utlity in HTD.
M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
  Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
  Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
  Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
  Rename
A b/src/main/protobuf/ClusterId.proto
  Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
  Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
  Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
  Added TableSchema and ColumnFamilySchema


This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15 
  src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3 
  src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af 
  src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059 
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
9e4ada9 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab 
  src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java 
PRE-CREATION 
  src/main/java/o

[jira] [Commented] (HBASE-6019) [refGuide] ported pseudo-distributed.html page to RefGuide Config chapter

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277271#comment-13277271
 ] 

Hudson commented on HBASE-6019:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #6 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/6/])
hbase-6019.  porting pseudo-dist html page to RefGuide Config chapter. 
(Revision 1339225)

 Result = FAILURE

> [refGuide] ported pseudo-distributed.html page to RefGuide Config chapter
> -
>
> Key: HBASE-6019
> URL: https://issues.apache.org/jira/browse/HBASE-6019
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: hbase_hbase_6019.patch
>
>
> Ported the separate pseudo-distributed.html page that existed outside the 
> RefGuie into the Configuration chapter.
> configuration.xml
> * Added an example of a local pseudo-dist HDFS config file (the refguide 
> didn't have this)
> * Ported pseudo-dist extras to pseudo-dist section.
> pseudo-distributed.xml
> * This is the old page.  I'm leaving this backward compatibility so that the 
> old links don't break, although the only thing this says now is the content 
> has been moved to the RefGuide
> site.xml
> * Removing pseudo-dist extras from the left-hand nav.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6026) Reconcile .proto files after the dust settles

[stack (JIRA)]

stack created HBASE-6026:


 Summary: Reconcile .proto files after the dust settles
 Key: HBASE-6026
 URL: https://issues.apache.org/jira/browse/HBASE-6026
 Project: HBase
  Issue Type: Task
Reporter: stack


Four or five of us have been working on .proto stuff over last few months.  
Andrew did pb work when we were all in diapers.  After HBASE-6000 goes in which 
moves all .protos into one place, lets look at doing reconciliation and 
cleanup.  For example, Andrew made a TableSchema that is also in stuff I added. 
 I did not use Andrews because it has stuff I do not want (though I took 
inspiration from his for what I did do) and ditto for a ColumnSchema -- in  
mine I call it ColumnFamilySchema.  After .proto resolution, we should try and 
reconcile method names for the common operations done (there was a bunch of 
overlap around our pb'ing work -- lets make sure we had same basic pattern all 
around so its easier on those that come after us figuring whats going on).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6025) Expose Hadoop Metrics through JSON Rest interface

[Elliott Clark (JIRA)]

Elliott Clark created HBASE-6025:


 Summary: Expose Hadoop Metrics through JSON Rest interface
 Key: HBASE-6025
 URL: https://issues.apache.org/jira/browse/HBASE-6025
 Project: HBase
  Issue Type: Improvement
Reporter: Elliott Clark
Assignee: Elliott Clark




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6023) Normalize security audit logging level with Hadoop

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277254#comment-13277254
 ] 

Andrew Purtell commented on HBASE-6023:
---

Will commit soon if no objection.

> Normalize security audit logging level with Hadoop
> --
>
> Key: HBASE-6023
> URL: https://issues.apache.org/jira/browse/HBASE-6023
> Project: HBase
>  Issue Type: Improvement
>  Components: ipc, security
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Attachments: 6023.patch
>
>
> A pretty trivial change, we log failed authentication attempts at WARN level, 
> as does Hadoop, but log successful authentication at TRACE level, where 
> Hadoop instead logs it at INFO level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6011) Unable to start master in local mode

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277253#comment-13277253
 ] 

Andrew Purtell commented on HBASE-6011:
---

Will commit v2 soon if no objection.

> Unable to start master in local mode
> 
>
> Key: HBASE-6011
> URL: https://issues.apache.org/jira/browse/HBASE-6011
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Attachments: 6011-v2.patch, 6011.patch
>
>
> Got this trying to launch head of 0.94 branch in local mode from the build 
> tree but it happens with trunk and 0.92 too:
> {noformat}
> 12/05/15 19:35:45 ERROR master.HMasterCommandLine: Failed to start master
> java.lang.ClassCastException: org.apache.hadoop.hbase.master.HMaster cannot 
> be cast to org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster
>   at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:142)
>   at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>   at 
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
>   at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1761)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6021) NullPointerException when running LoadTestTool without specifying compression type

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277252#comment-13277252
 ] 

Andrew Purtell commented on HBASE-6021:
---

Will commit this soon if no objections.

> NullPointerException when running LoadTestTool without specifying compression 
> type
> --
>
> Key: HBASE-6021
> URL: https://issues.apache.org/jira/browse/HBASE-6021
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.96.0, 0.94.1
> Environment: Hadoop 1.0.2, HBase 0.94.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Attachments: 6021.patch
>
>
> If you don't specify a compression type on the LoadTestTool command line then 
> this happens:
> {noformat}
> 12/05/16 18:41:23 ERROR util.AbstractHBaseTool: Error running command-line 
> tool
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.HColumnDescriptor.setCompressionType(HColumnDescriptor.java:535)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.createPreSplitLoadTestTable(HBaseTestingUtility.java:1885)
>   at 
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:297)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:103)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:173)
>   at org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:341)
> {noformat}
> This should be handled better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6021) NullPointerException when running LoadTestTool without specifying compression type

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277242#comment-13277242
 ] 

Hadoop QA commented on HBASE-6021:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527728/6021.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 31 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
  org.apache.hadoop.hbase.master.TestSplitLogManager

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1900//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1900//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1900//console

This message is automatically generated.

> NullPointerException when running LoadTestTool without specifying compression 
> type
> --
>
> Key: HBASE-6021
> URL: https://issues.apache.org/jira/browse/HBASE-6021
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.96.0, 0.94.1
> Environment: Hadoop 1.0.2, HBase 0.94.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Attachments: 6021.patch
>
>
> If you don't specify a compression type on the LoadTestTool command line then 
> this happens:
> {noformat}
> 12/05/16 18:41:23 ERROR util.AbstractHBaseTool: Error running command-line 
> tool
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.HColumnDescriptor.setCompressionType(HColumnDescriptor.java:535)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.createPreSplitLoadTestTable(HBaseTestingUtility.java:1885)
>   at 
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:297)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:103)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:173)
>   at org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:341)
> {noformat}
> This should be handled better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277237#comment-13277237
 ] 

Lars Hofhansl commented on HBASE-6018:
--

ThreadPoolExecutor is pretty stupid (if you ask me). Unless the Queue fills up 
it will never allocate more than threads than indicated by the number of core 
threads.
The LinkedBlockingQueue has no limit, so it'll only ever use 
conf.getInt("hbasefsck.numthreads") number of threads.

> hbck fails with a RejectedExecutionException
> 
>
> Key: HBASE-6018
> URL: https://issues.apache.org/jira/browse/HBASE-6018
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.94.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-6018.patch
>
>
> On a long running job 0.94.0rc3 cluster, we get to a point where hbck 
> consistently encounters this error and fails:
> {code}
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
>   at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
>   at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>   at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>   at 
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
>   at 
> org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
>   at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>   at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277236#comment-13277236
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--



bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 473
bq.  > 
bq.  >
bq.  > Why you remove this?  We don't return these any more?  Offline I 
think is 'dead', unused now.  Split not.

If we discover that a region has been split in META, than, it is past 
point-of-no-return and the region cannot be seen un-split anymore, even though 
concurrent rs failures.  for getStartEndKeys() we are returning whatever the 
getRegionLocation() provides. getRegionLocations() ignores offline regions, but 
returns daughter regions for split-parents (which are offline as well).


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 351
bq.  > 
bq.  >
bq.  > Should this be public?  Should it remain internal to the HTable 
hidden?

MetaScanner is confusing in its visibility. It's javadoc states "Although 
public visibility, this is not a public-facing API and may evolve in minor 
releases.", but it is annotated with @InterfaceAudience.Public 
@InterfaceStability.Evolving. I think for BlockingMetaScannerVisitor, we should 
set the visibility the same as MetaScannerVisitor. I think, MetaScannerVisitor 
itself is of little use without the BlockingMetaScannerVisitor functionality 
due to this issue. 
Shall we change MetaScanner, MSV and BMSV to be @InterfaceAudience.Private, 
wdyt? 


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 310
bq.  > 
bq.  >
bq.  > Bit of javadoc to say this is best-effort.
bq.  > 
bq.  > Also, does this belong in MetaReader (Won't hold you to it... these 
two classes, a MetaReader vs MetaEditor are kinda silly... this whole catalog 
package needs killing).

agreed


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 395
bq.  > 
bq.  >
bq.  > Yeah, so, what happens if daughter has split by the time I get here?

The scanner provides the regions results in sorted order, and since the 
daughters are sorted after the parent, we always process parent first. When we 
process the daughter regions manually (processRow(resultA)), we also add them 
to the daughterRegions set. If the scanner also sees them, they are just 
skipped (line 373)


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 434
bq.  > 
bq.  >
bq.  > So if interrupted or we don't find it by the time the blocking time 
has passed, we just return null?   What you reckon?  We should at least 
complain?

yeah, I think we can throw RegionOfflineException upon timeout and interrupt. 
The operation can be retried by the client upon timeout.


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 465
bq.  > 
bq.  >
bq.  > Does the scan of meta start at first table region?

MetaScanner.metaScan() ensures that the scan starts at the first table region. 


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 390
bq.  > 
bq.  >
bq.  > Yeah, what Ted says... Can you close when done?  See 
MetaEditor/MetaReader.  They do this a bunch.  Closing means for sure the zk 
and connection resources will be cleaned up afterward.  Its reference counting 
so keepign around an HTable could mess it up.

We are already closing the HTable's. But let me see how we can best reuse 
Htable instances. 


- enis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7939
---


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq

[jira] [Updated] (HBASE-5953) Expose the current state of the balancerSwitch

[David S. Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang updated HBASE-5953:
-

Affects Version/s: (was: 0.92.1)
   (was: 0.94.0)
 Assignee: Gregory Chanan  (was: David S. Wang)

Assigning to Greg at his request.  Removing 0.92 and 0.94 from fixVersions at 
least until we have a compatibility story for ClusterStatus (see HBASE-6009 for 
details).

> Expose the current state of the balancerSwitch
> --
>
> Key: HBASE-5953
> URL: https://issues.apache.org/jira/browse/HBASE-5953
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.96.0
>Reporter: David S. Wang
>Assignee: Gregory Chanan
>Priority: Minor
>
> The balancerSwitch as in both 0.90 and 0.92.1 is implemented in such a way 
> that it is impossible to retrieve its value without changing it:
> /**
> Turn the load balancer on or off.
> @param b If true, enable balancer. If false, disable balancer.
> @return Previous balancer value
> */
> public boolean balanceSwitch(final boolean b);
> It would be nice to have a way to just get the current state.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6023) Normalize security audit logging level with Hadoop

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277226#comment-13277226
 ] 

Hadoop QA commented on HBASE-6023:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527723/6023.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 31 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1899//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1899//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1899//console

This message is automatically generated.

> Normalize security audit logging level with Hadoop
> --
>
> Key: HBASE-6023
> URL: https://issues.apache.org/jira/browse/HBASE-6023
> Project: HBase
>  Issue Type: Improvement
>  Components: ipc, security
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Attachments: 6023.patch
>
>
> A pretty trivial change, we log failed authentication attempts at WARN level, 
> as does Hadoop, but log successful authentication at TRACE level, where 
> Hadoop instead logs it at INFO level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-6024) Add state of load balancer to cluster status

[David S. Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David S. Wang resolved HBASE-6024.
--

Resolution: Duplicate

Duplicate of HBASE-5953 as Ted stated.

> Add state of load balancer to cluster status
> 
>
> Key: HBASE-6024
> URL: https://issues.apache.org/jira/browse/HBASE-6024
> Project: HBase
>  Issue Type: Task
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.96.0
>
>
> There currently does not seem to be anyway to see if the load balancer is 
> actually running or not.  synchronousBalanceSwitch and balanceSwitch will 
> return the state, but require possibly turning off/on the balancer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5987) HFileBlockIndex improvement

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5987:
--

Status: Open  (was: Patch Available)

I forgot the patch needs to be ported to trunk.

> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, 
> D3237.4.patch, D3237.5.patch, D3237.6.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value for each data block and the target 
> key value is beween the last key value of data block N-1 and data block N, 
> then the target key value is supposed be data block N for sure. 
> As long as HBase only supports the forward scan, the last key value makes 
> more sense to be indexed on than the start key value. 
> Thanks Kannan and Mikhail for the insightful discussions and suggestions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-5999) AggregationClient throws an exception when startRow is set and stopRow is not set in scan object.

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu resolved HBASE-5999.
---

Resolution: Not A Problem

This has been fixed in 0.92, etc

> AggregationClient throws an exception when startRow is set and stopRow is not 
> set in scan object.
> -
>
> Key: HBASE-5999
> URL: https://issues.apache.org/jira/browse/HBASE-5999
> Project: HBase
>  Issue Type: Bug
>  Components: client, coprocessors
>Affects Versions: 0.92.0, 0.92.1
>Reporter: Anil Gupta
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> AggregationClient throws an exception when the startRow is set and stopRow is 
> not set in scan object. AggregationClient should not throw the exception in 
> this case because the user might want to scan the entire table starting from 
> the startRow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6024) Add state of load balancer to cluster status

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277220#comment-13277220
 ] 

Zhihong Yu commented on HBASE-6024:
---

This seems to be dup of HBASE-5953.

> Add state of load balancer to cluster status
> 
>
> Key: HBASE-6024
> URL: https://issues.apache.org/jira/browse/HBASE-6024
> Project: HBase
>  Issue Type: Task
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.96.0
>
>
> There currently does not seem to be anyway to see if the load balancer is 
> actually running or not.  synchronousBalanceSwitch and balanceSwitch will 
> return the state, but require possibly turning off/on the balancer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277218#comment-13277218
 ] 

Hadoop QA commented on HBASE-5987:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527710/D3237.6.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 14 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1902//console

This message is automatically generated.

> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, 
> D3237.4.patch, D3237.5.patch, D3237.6.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value for each data block and the target 
> key value is beween the last key value of data block N-1 and data block N, 
> then the target key value is supposed be data block N for sure. 
> As long as HBase only supports the forward scan, the last key value makes 
> more sense to be indexed on than the start key value. 
> Thanks Kannan and Mikhail for the insightful discussions and suggestions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277217#comment-13277217
 ] 

Hadoop QA commented on HBASE-5959:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12527726/HBASE-5959.D3189.5.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 16 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1901//console

This message is automatically generated.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6024) Add state of load balancer to cluster status

[Gregory Chanan (JIRA)]

Gregory Chanan created HBASE-6024:
-

 Summary: Add state of load balancer to cluster status
 Key: HBASE-6024
 URL: https://issues.apache.org/jira/browse/HBASE-6024
 Project: HBase
  Issue Type: Task
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
 Fix For: 0.96.0


There currently does not seem to be anyway to see if the load balancer is 
actually running or not.  synchronousBalanceSwitch and balanceSwitch will 
return the state, but require possibly turning off/on the balancer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277214#comment-13277214
 ] 

Lars Hofhansl commented on HBASE-5986:
--

As a general comment I think it would be better if instead of blocking on the 
server, HTable.getStartEndKeys() could detect the inconsistency and try again.

> Clients can see holes in the META table when regions are being split
> 
>
> Key: HBASE-5986
> URL: https://issues.apache.org/jira/browse/HBASE-5986
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.1, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-5986-test_v1.patch
>
>
> We found this issue when running large scale ingestion tests for HBASE-5754. 
> The problem is that the .META. table updates are not atomic while splitting a 
> region. In SplitTransaction, there is a time lap between the marking the 
> parent offline, and adding of daughters to the META table. This can result in 
> clients using MetaScanner, of HTable.getStartEndKeys (used by the 
> TableInputFormat) missing regions which are made just offline, but the 
> daughters are not added yet. 
> This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6004) Adding more logging to help debugging MR job

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277213#comment-13277213
 ] 

Hadoop QA commented on HBASE-6004:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527719/hbase-6004_v2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 31 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1898//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1898//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1898//console

This message is automatically generated.

> Adding more logging to help debugging MR job
> 
>
> Key: HBASE-6004
> URL: https://issues.apache.org/jira/browse/HBASE-6004
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.94.0, 0.96.0
>
> Attachments: hbase-6004.patch, hbase-6004_v2-0.94.patch, 
> hbase-6004_v2.patch
>
>
> MR job sometime fails because scanner expired. In this case, it will be 
> helpful to know the last successful row, the ip of the region sever, and so 
> on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5987) HFileBlockIndex improvement

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5987:
--

Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, D3237.2.patch, D3237.3.patch, 
> D3237.4.patch, D3237.5.patch, D3237.6.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value for each data block and the target 
> key value is beween the last key value of data block N-1 and data block N, 
> then the target key value is supposed be data block N for sure. 
> As long as HBase only supports the forward scan, the last key value makes 
> more sense to be indexed on than the start key value. 
> Thanks Kannan and Mikhail for the insightful discussions and suggestions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-5826) Improve sync of HLog edits

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu reassigned HBASE-5826:
-

Assignee: Todd Lipcon

> Improve sync of HLog edits
> --
>
> Key: HBASE-5826
> URL: https://issues.apache.org/jira/browse/HBASE-5826
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zhihong Yu
>Assignee: Todd Lipcon
> Fix For: 0.96.0
>
> Attachments: 5826-v2.txt, 5826-v3.txt, 5826.txt
>
>
> HBASE-5782 solved the correctness issue for the sync of HLog edits.
> Todd provided a patch that would achieve higher throughput.
> This JIRA is a continuation of Todd's work submitted there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5826) Improve sync of HLog edits

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277209#comment-13277209
 ] 

Zhihong Yu commented on HBASE-5826:
---

TestAdmin#testCloseRegionThatFetchesTheHRIFromMeta failure wasn't related to 
the patch.

Locally it passed for me.

Are there further review comments about Todd's patch ?

> Improve sync of HLog edits
> --
>
> Key: HBASE-5826
> URL: https://issues.apache.org/jira/browse/HBASE-5826
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 5826-v2.txt, 5826-v3.txt, 5826.txt
>
>
> HBASE-5782 solved the correctness issue for the sync of HLog edits.
> Todd provided a patch that would achieve higher throughput.
> This JIRA is a continuation of Todd's work submitted there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277206#comment-13277206
 ] 

Lars Hofhansl commented on HBASE-5954:
--

I'm using ext4 with default mount options (indeed the numbers would be useless 
otherwise)

> Allow proper fsync support for HBase
> 
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 
> 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5826) Improve sync of HLog edits

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277204#comment-13277204
 ] 

Hadoop QA commented on HBASE-5826:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527713/5826-v3.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 31 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.client.TestAdmin

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1896//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1896//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1896//console

This message is automatically generated.

> Improve sync of HLog edits
> --
>
> Key: HBASE-5826
> URL: https://issues.apache.org/jira/browse/HBASE-5826
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zhihong Yu
> Fix For: 0.96.0
>
> Attachments: 5826-v2.txt, 5826-v3.txt, 5826.txt
>
>
> HBASE-5782 solved the correctness issue for the sync of HLog edits.
> Todd provided a patch that would achieve higher throughput.
> This JIRA is a continuation of Todd's work submitted there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6021) NullPointerException when running LoadTestTool without specifying compression type

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-6021:
--

Attachment: 6021.patch

Trivial patch.

> NullPointerException when running LoadTestTool without specifying compression 
> type
> --
>
> Key: HBASE-6021
> URL: https://issues.apache.org/jira/browse/HBASE-6021
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.96.0, 0.94.1
> Environment: Hadoop 1.0.2, HBase 0.94.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Attachments: 6021.patch
>
>
> If you don't specify a compression type on the LoadTestTool command line then 
> this happens:
> {noformat}
> 12/05/16 18:41:23 ERROR util.AbstractHBaseTool: Error running command-line 
> tool
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.HColumnDescriptor.setCompressionType(HColumnDescriptor.java:535)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.createPreSplitLoadTestTable(HBaseTestingUtility.java:1885)
>   at 
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:297)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:103)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:173)
>   at org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:341)
> {noformat}
> This should be handled better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6021) NullPointerException when running LoadTestTool without specifying compression type

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-6021:
--

Status: Patch Available  (was: Open)

> NullPointerException when running LoadTestTool without specifying compression 
> type
> --
>
> Key: HBASE-6021
> URL: https://issues.apache.org/jira/browse/HBASE-6021
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.96.0, 0.94.1
> Environment: Hadoop 1.0.2, HBase 0.94.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Attachments: 6021.patch
>
>
> If you don't specify a compression type on the LoadTestTool command line then 
> this happens:
> {noformat}
> 12/05/16 18:41:23 ERROR util.AbstractHBaseTool: Error running command-line 
> tool
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.HColumnDescriptor.setCompressionType(HColumnDescriptor.java:535)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.createPreSplitLoadTestTable(HBaseTestingUtility.java:1885)
>   at 
> org.apache.hadoop.hbase.util.LoadTestTool.doWork(LoadTestTool.java:297)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:103)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:173)
>   at org.apache.hadoop.hbase.util.LoadTestTool.main(LoadTestTool.java:341)
> {noformat}
> This should be handled better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277200#comment-13277200
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7944
---


In a previous comment you said (about the HTableDescriptor/HColumnDesriptor pb 
stuff):
"Well, Andrew beat us both to it over in his REST pb stuff.  We need to 
reconcile his w/ ours too"

Where is his stuff?  I couldn't find it.  Should we create a JIRA about 
reconciling? It would be nice to have something, however imperfect, up in trunk 
to work against, then we could fix up later.


src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java


This isn't ever read?



src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java


This constructor sets pbMade to false, but it should be true in this case, 
right?


- Gregory


On 2012-05-16 17:02:35, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 17:02:35)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq.Range in Reference is no longer public.
bq.Range in Reference is no longer public.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq.ClusterId got renamed ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq.Use new serialization utlity in HTD.
bq.  M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq.Generic method for writing dot file content.
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Reference#Range is not public any more
bq.  M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq.Deprecated getHRegionInfo, etc.
bq.  D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq.  A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq.Rename
bq.  A b/src/main/protobuf/ClusterId.proto
bq.Added file for ClusterId only since its written to fs and to zk.
bq.  A b/src/main/protobuf/FS.proto
bq.Protos for fs files.
bq.  M b/src/main/protobuf/ZooKeeper.proto
bq.Moved Clu

[jira] [Updated] (HBASE-5959) Add other load balancers

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5959:
---

Attachment: HBASE-5959.D3189.5.patch

eclark updated the revision "HBASE-5959 [jira] Add other load balancers".
Reviewers: JIRA

  More debug output and some small wording/spelling


REVISION DETAIL
  https://reviews.facebook.net/D3189

AFFECTED FILES
  pom.xml
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
  src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
  src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
  src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
  
src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
  src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
  
src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
  
src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
  src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
  src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
  src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
  src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
  
src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
  
src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
  
src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

To: JIRA, eclark
Cc: tedyu


> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch, HBASE-5959.D3189.5.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6023) Normalize security audit logging level with Hadoop

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-6023:
--

Attachment: 6023.patch

> Normalize security audit logging level with Hadoop
> --
>
> Key: HBASE-6023
> URL: https://issues.apache.org/jira/browse/HBASE-6023
> Project: HBase
>  Issue Type: Improvement
>  Components: ipc, security
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Attachments: 6023.patch
>
>
> A pretty trivial change, we log failed authentication attempts at WARN level, 
> as does Hadoop, but log successful authentication at TRACE level, where 
> Hadoop instead logs it at INFO level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6023) Normalize security audit logging level with Hadoop

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-6023:
--

Status: Patch Available  (was: Open)

> Normalize security audit logging level with Hadoop
> --
>
> Key: HBASE-6023
> URL: https://issues.apache.org/jira/browse/HBASE-6023
> Project: HBase
>  Issue Type: Improvement
>  Components: ipc, security
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Attachments: 6023.patch
>
>
> A pretty trivial change, we log failed authentication attempts at WARN level, 
> as does Hadoop, but log successful authentication at TRACE level, where 
> Hadoop instead logs it at INFO level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5926) Delete the master znode after a master crash

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277190#comment-13277190
 ] 

Zhihong Yu commented on HBASE-5926:
---

{code}
+"Usage: Master [opts] start|stop|cleanZNode\n" +
{code}
Please add document for cleanZNode command.
{code}
+   * delete the znode master if its content is same to the parameter
{code}
'znode master' -> 'master znode', 'same to' -> 'same as'
{code}
+} catch (KeeperException ignore) {
+} catch (IOException ignore) {
+}
{code}
I would expect some logging for above cases.


> Delete the master znode after a master crash
> 
>
> Key: HBASE-5926
> URL: https://issues.apache.org/jira/browse/HBASE-5926
> Project: HBase
>  Issue Type: Improvement
>  Components: master, scripts
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5926.v6.patch
>
>
> This is the continuation of the work done in HBASE-5844.
> But we can't apply exactly the same strategy: for the region server, there is 
> a znode per region server, while for the master & backup master there is a 
> single znode for both.
> So if we apply the same strategy as for a regionserver, we may have this 
> scenario:
> 1) Master starts
> 2) Backup master starts
> 3) Master dies
> 4) ZK detects it
> 5) Backup master receives the update from ZK
> 6) Backup master creates the new master node and become the main master
> 7) Previous master script continues
> 8) Previous master script deletes the master node in ZK
> 9) => issue: we deleted the node just created by the new master
> This should not happen often (usually the znode will be deleted soon enough), 
> but it can happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6023) Normalize security audit logging level with Hadoop

[Andrew Purtell (JIRA)]

Andrew Purtell created HBASE-6023:
-

 Summary: Normalize security audit logging level with Hadoop
 Key: HBASE-6023
 URL: https://issues.apache.org/jira/browse/HBASE-6023
 Project: HBase
  Issue Type: Improvement
  Components: ipc, security
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor


A pretty trivial change, we log failed authentication attempts at WARN level, 
as does Hadoop, but log successful authentication at TRACE level, where Hadoop 
instead logs it at INFO level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5104) Provide a reliable intra-row pagination mechanism

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277188#comment-13277188
 ] 

Phabricator commented on HBASE-5104:


madhuvaidya has commented on the revision "[jira] [HBASE-5104] Provide a 
reliable intra-row pagination mechanism".

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/client/Get.java:472 This was done to 
maintain inter-op if we are not using either the storeLimit or the storeOffset.

REVISION DETAIL
  https://reviews.facebook.net/D2799

To: madhuvaidya, lhofhansl, Kannan, tedyu, stack, todd, JIRA, jxcn01, mbautin
Cc: jxcn01, Liyin


> Provide a reliable intra-row pagination mechanism
> -
>
> Key: HBASE-5104
> URL: https://issues.apache.org/jira/browse/HBASE-5104
> Project: HBase
>  Issue Type: Bug
>Reporter: Kannan Muthukkaruppan
>Assignee: Madhuwanti Vaidya
> Attachments: D2799.1.patch, D2799.2.patch, D2799.3.patch, 
> D2799.4.patch, D2799.5.patch, D2799.6.patch, 
> jira-HBASE-5104-Provide-a-reliable-intra-row-paginat-2012-04-16_12_39_42.patch,
>  testFilterList.rb
>
>
> Addendum:
> Doing pagination (retrieving at most "limit" number of KVs at a particular 
> "offset") is currently supported via the ColumnPaginationFilter. However, it 
> is not a very clean way of supporting pagination.  Some of the problems with 
> it are:
> * Normally, one would expect a query with (Filter(A) AND Filter(B)) to have 
> same results as (query with Filter(A)) INTERSECT (query with Filter(B)). This 
> is not the case for ColumnPaginationFilter as its internal state gets updated 
> depending on whether or not Filter(A) returns TRUE/FALSE for a particular 
> cell.
> * When this Filter is used in combination with other filters (e.g., doing AND 
> with another filter using FilterList), the behavior of the query depends on 
> the order of filters in the FilterList. This is not ideal.
> * ColumnPaginationFilter is a stateful filter which ends up counting multiple 
> versions of the cell as separate values even if another filter upstream or 
> the ScanQueryMatcher is going to reject the value for other reasons.
> Seems like we need a reliable way to do pagination. The particular use case 
> that prompted this JIRA is pagination within the same rowKey. For example, 
> for a given row key R, get columns with prefix P, starting at offset X (among 
> columns which have prefix P) and limit Y. Some possible fixes might be:
> 1) enhance ColumnPrefixFilter to support another constructor which supports 
> limit/offset.
> 2) Support pagination (limit/offset) at the Scan/Get API level (rather than 
> as a filter) [Like SQL].
> Original Post:
> Thanks Jiakai Liu for reporting this issue and doing the initial 
> investigation. Email from Jiakai below:
> Assuming that we have an index column family with the following entries:
> "tag0:001:thread1"
> ...
> "tag1:001:thread1"
> "tag1:002:thread2"
> ...
> "tag1:010:thread10"
> ...
> "tag2:001:thread1"
> "tag2:005:thread5"
> ...
> To get threads with "tag1" in range [5, 10), I tried the following code:
> ColumnPrefixFilter filter1 = new 
> ColumnPrefixFilter(Bytes.toBytes("tag1"));
> ColumnPaginationFilter filter2 = new ColumnPaginationFilter(5 /* limit 
> */, 5 /* offset */);
> FilterList filters = new FilterList(Operator.MUST_PASS_ALL);
> filters.addFilter(filter1);
> filters.addFilter(filter2);
> Get get = new Get(USER);
> get.addFamily(COLUMN_FAMILY);
> get.setMaxVersions(1);
> get.setFilter(filters);
> Somehow it didn't work as expected. It returned the entries as if the filter1 
> were not set.
> Turns out the ColumnPrefixFilter returns SEEK_NEXT_USING_HINT in some cases. 
> The FilterList filter does not handle this return code properly (treat it as 
> INCLUDE).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5104) Provide a reliable intra-row pagination mechanism

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277182#comment-13277182
 ] 

Phabricator commented on HBASE-5104:


stack has commented on the revision "[jira] [HBASE-5104] Provide a reliable 
intra-row pagination mechanism".

  lgtm

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/client/Get.java:212 Will this be 
accurate if rows are inserted meantime (or deleted?).
  src/main/java/org/apache/hadoop/hbase/client/Get.java:201 This is great.  One 
day we should do size-based too.
  src/main/java/org/apache/hadoop/hbase/client/Get.java:472 Why not just write 
out our version as 3?  To save some bytes on wire?
  src/main/java/org/apache/hadoop/hbase/client/Scan.java:102 Doesn't Scan and 
Get share common ancestor?
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java:647 THanks 
for doing this.
  
src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java:452
 You need to add below to each classified test for classification to work

@org.junit.Rule
public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu =
  new org.apache.hadoop.hbase.ResourceCheckerJUnitRule();

REVISION DETAIL
  https://reviews.facebook.net/D2799

To: madhuvaidya, lhofhansl, Kannan, tedyu, stack, todd, JIRA, jxcn01, mbautin
Cc: jxcn01, Liyin


> Provide a reliable intra-row pagination mechanism
> -
>
> Key: HBASE-5104
> URL: https://issues.apache.org/jira/browse/HBASE-5104
> Project: HBase
>  Issue Type: Bug
>Reporter: Kannan Muthukkaruppan
>Assignee: Madhuwanti Vaidya
> Attachments: D2799.1.patch, D2799.2.patch, D2799.3.patch, 
> D2799.4.patch, D2799.5.patch, D2799.6.patch, 
> jira-HBASE-5104-Provide-a-reliable-intra-row-paginat-2012-04-16_12_39_42.patch,
>  testFilterList.rb
>
>
> Addendum:
> Doing pagination (retrieving at most "limit" number of KVs at a particular 
> "offset") is currently supported via the ColumnPaginationFilter. However, it 
> is not a very clean way of supporting pagination.  Some of the problems with 
> it are:
> * Normally, one would expect a query with (Filter(A) AND Filter(B)) to have 
> same results as (query with Filter(A)) INTERSECT (query with Filter(B)). This 
> is not the case for ColumnPaginationFilter as its internal state gets updated 
> depending on whether or not Filter(A) returns TRUE/FALSE for a particular 
> cell.
> * When this Filter is used in combination with other filters (e.g., doing AND 
> with another filter using FilterList), the behavior of the query depends on 
> the order of filters in the FilterList. This is not ideal.
> * ColumnPaginationFilter is a stateful filter which ends up counting multiple 
> versions of the cell as separate values even if another filter upstream or 
> the ScanQueryMatcher is going to reject the value for other reasons.
> Seems like we need a reliable way to do pagination. The particular use case 
> that prompted this JIRA is pagination within the same rowKey. For example, 
> for a given row key R, get columns with prefix P, starting at offset X (among 
> columns which have prefix P) and limit Y. Some possible fixes might be:
> 1) enhance ColumnPrefixFilter to support another constructor which supports 
> limit/offset.
> 2) Support pagination (limit/offset) at the Scan/Get API level (rather than 
> as a filter) [Like SQL].
> Original Post:
> Thanks Jiakai Liu for reporting this issue and doing the initial 
> investigation. Email from Jiakai below:
> Assuming that we have an index column family with the following entries:
> "tag0:001:thread1"
> ...
> "tag1:001:thread1"
> "tag1:002:thread2"
> ...
> "tag1:010:thread10"
> ...
> "tag2:001:thread1"
> "tag2:005:thread5"
> ...
> To get threads with "tag1" in range [5, 10), I tried the following code:
> ColumnPrefixFilter filter1 = new 
> ColumnPrefixFilter(Bytes.toBytes("tag1"));
> ColumnPaginationFilter filter2 = new ColumnPaginationFilter(5 /* limit 
> */, 5 /* offset */);
> FilterList filters = new FilterList(Operator.MUST_PASS_ALL);
> filters.addFilter(filter1);
> filters.addFilter(filter2);
> Get get = new Get(USER);
> get.addFamily(COLUMN_FAMILY);
> get.setMaxVersions(1);
> get.setFilter(filters);
> Somehow it didn't work as expected. It returned the entries as if the filter1 
> were not set.
> Turns out the ColumnPrefixFilter returns SEEK_NEXT_USING_HINT in some cases. 
> The FilterList filter does not handle this return code properly (treat it as 
> INCLUDE).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5926) Delete the master znode after a master crash

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277180#comment-13277180
 ] 

Zhihong Yu commented on HBASE-5926:
---

I cannot find CleanZnode class in the patch.

> Delete the master znode after a master crash
> 
>
> Key: HBASE-5926
> URL: https://issues.apache.org/jira/browse/HBASE-5926
> Project: HBase
>  Issue Type: Improvement
>  Components: master, scripts
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5926.v6.patch
>
>
> This is the continuation of the work done in HBASE-5844.
> But we can't apply exactly the same strategy: for the region server, there is 
> a znode per region server, while for the master & backup master there is a 
> single znode for both.
> So if we apply the same strategy as for a regionserver, we may have this 
> scenario:
> 1) Master starts
> 2) Backup master starts
> 3) Master dies
> 4) ZK detects it
> 5) Backup master receives the update from ZK
> 6) Backup master creates the new master node and become the main master
> 7) Previous master script continues
> 8) Previous master script deletes the master node in ZK
> 9) => issue: we deleted the node just created by the new master
> This should not happen often (usually the znode will be deleted soon enough), 
> but it can happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6004) Adding more logging to help debugging MR job

[Jimmy Xiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6004:
---

Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

> Adding more logging to help debugging MR job
> 
>
> Key: HBASE-6004
> URL: https://issues.apache.org/jira/browse/HBASE-6004
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.96.0, 0.94.0
>
> Attachments: hbase-6004.patch, hbase-6004_v2-0.94.patch, 
> hbase-6004_v2.patch
>
>
> MR job sometime fails because scanner expired. In this case, it will be 
> helpful to know the last successful row, the ip of the region sever, and so 
> on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6004) Adding more logging to help debugging MR job

[Jimmy Xiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6004:
---

Attachment: hbase-6004_v2.patch

> Adding more logging to help debugging MR job
> 
>
> Key: HBASE-6004
> URL: https://issues.apache.org/jira/browse/HBASE-6004
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.94.0, 0.96.0
>
> Attachments: hbase-6004.patch, hbase-6004_v2-0.94.patch, 
> hbase-6004_v2.patch
>
>
> MR job sometime fails because scanner expired. In this case, it will be 
> helpful to know the last successful row, the ip of the region sever, and so 
> on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5926) Delete the master znode after a master crash

[nkeywal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277172#comment-13277172
 ] 

nkeywal commented on HBASE-5926:


the race condition is decreased to a production-acceptable minimum imho. We do 
a compare & delete in the java code, so the race condition is now: between the 
comparison and the delete, we fail if, and only if: the session expires and the 
master node is deleted and the master backup recreates the node. That's 
unlikely. 

> Delete the master znode after a master crash
> 
>
> Key: HBASE-5926
> URL: https://issues.apache.org/jira/browse/HBASE-5926
> Project: HBase
>  Issue Type: Improvement
>  Components: master, scripts
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5926.v6.patch
>
>
> This is the continuation of the work done in HBASE-5844.
> But we can't apply exactly the same strategy: for the region server, there is 
> a znode per region server, while for the master & backup master there is a 
> single znode for both.
> So if we apply the same strategy as for a regionserver, we may have this 
> scenario:
> 1) Master starts
> 2) Backup master starts
> 3) Master dies
> 4) ZK detects it
> 5) Backup master receives the update from ZK
> 6) Backup master creates the new master node and become the main master
> 7) Previous master script continues
> 8) Previous master script deletes the master node in ZK
> 9) => issue: we deleted the node just created by the new master
> This should not happen often (usually the znode will be deleted soon enough), 
> but it can happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira