[jira] [Updated] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN

[fulin wang (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

fulin wang updated HBASE-5599:
--

Attachment: hbase-5599-0.90_v6.patch

I make a hbase-5599-0.90_v6.patch, please review.
This is test result:
Tests Errors  Failures Skipped Success Rate Time 
14 0 0 0 100% 181.667 

TestHBaseFsck testHBaseFsckClean 4.843 
 testDupeStartKey 10.086 
 testDupeRegion 10.382 
 testDegenerateRegions 10.098 
 testContainedRegionOverlap 9.598 
 testOverlapAndOrphan 12.687 
 testCoveredStartKey 10.836 
 testRegionHole 8.652 
 testHDFSRegioninfoMissing 11.895 
 testNotInMetaOrDeployedHole 8.262 
 testNotInMetaHole 8.387 
 testNotInHdfs 8.345 
 testNoHdfsTable 46.275

> The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, 
> NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, 
> FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN.
> 
>
> Key: HBASE-5599
> URL: https://issues.apache.org/jira/browse/HBASE-5599
> Project: HBase
>  Issue Type: New Feature
>  Components: hbck
>Affects Versions: 0.90.6
>Reporter: fulin wang
> Fix For: 0.90.6
>
> Attachments: hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, 
> hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, hbase-5599-0.90_v6.patch, 
> hbase-5599-0.92_v5.patch, hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch
>
>
> The hbck tool can not fix the six scenarios.
> 1. Version file does not exist in root dir.
>Fix: I try to create a version file by 'FSUtils.setVersion' method.
>
> 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any 
> region server.
>Fix: I get region info form the hdfs file, this region info write to 
> '.META.' table.
>
> 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
>Fix: I get region info form the hdfs file, this region info write to 
> '.META.' table.
>
> 4. [REGIONNAME] should not be deployed according to META, but is deployed on 
> [SERVERNAME]
>Fix: Close this region.
>
> 5. First region should start with an empty key.  You need to  create a new 
> region and regioninfo in HDFS to plug the hole.
>Fix: The region info is not in hdfs and .META., so it create a empty 
> region for this error.
> 6. There is a hole in the region chain between [KEY] and [KEY]. You need to 
> create a new regioninfo and region dir in hdfs to plug the hole.
>   Fix: The region info is not in hdfs and .META., so it create a empty region 
> for this hole.
>   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5673) The OOM problem of IPC client call cause all handle block

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242112#comment-13242112
 ] 

Hudson commented on HBASE-5673:
---

Integrated in HBase-0.94 #69 (See 
[https://builds.apache.org/job/HBase-0.94/69/])
HBASE-5673 The OOM problem of IPC client call cause all handle block; 
REAPPLY; V2 OF PATCH (Revision 1307276)
HBASE-5673 The OOM problem of IPC client call cause all handle block; REVERT -- 
APPLIED WRONG PATCH (Revision 1307271)

 Result = FAILURE
stack : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java

stack : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java


> The OOM problem of IPC client call  cause all handle block
> --
>
> Key: HBASE-5673
> URL: https://issues.apache.org/jira/browse/HBASE-5673
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
> Environment: 0.90.6
>Reporter: xufeng
>Assignee: xufeng
> Fix For: 0.90.7, 0.92.2, 0.94.1
>
> Attachments: HBASE-5673-90-V2.patch, HBASE-5673-90.patch
>
>
> if HBaseClient meet "unable to create new native thread" exception, the call 
> will never complete because it be lost in calls queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5677) The master never does balance because duplicate openhandled the one region

[xufeng (Created) (JIRA)]

The master never does balance because duplicate openhandled the one region
--

 Key: HBASE-5677
 URL: https://issues.apache.org/jira/browse/HBASE-5677
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.6
 Environment: 0.90
Reporter: xufeng
Assignee: xufeng


If region be assigned When the master is doing initialization（before do 
processFailover）,the region will be duplicate openhandled.
it cause the region in RIT,thus the master never does balance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5674) add support in HBase to overwrite hbase timestamp to a version number during major compaction

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242102#comment-13242102
 ] 

stack commented on HBASE-5674:
--

I didn't get the 'reference'.  Sorry, that email went over my head.

bq. So are you referring this as conflicting with your 'hardcore production 
worthy platform' goal?

First its not 'my' goal.   Check out the notes from recent HBase PMC meeting:  
http://blogs.apache.org/hbase/.

That said, I thought it a different kinda 'researchy' that was being referred 
to.  I like the 'research' you fellas are at.

(Pardon me.  I did not read the name on the issue before responding.  All I saw 
was the short description asking for an 'odd', little-substantiated behavior 
and I asked a question.  Even on your comeback, I failed to check the whom and 
was innocently reacting to what I thought was a request that hbase become a 
dumping ground for plugins and research)

 

> add support in HBase to overwrite hbase timestamp to a version number during 
> major compaction
> -
>
> Key: HBASE-5674
> URL: https://issues.apache.org/jira/browse/HBASE-5674
> Project: HBase
>  Issue Type: Improvement
>Reporter: He Yongqiang
>Assignee: He Yongqiang
>
> Right now, a millisecond-level timestamp is attached to every record. 
> In our case, we only need a version number (mostly it will be just zero etc). 
> A millisecond timestamp is too heavy to carry. We should add support to 
> overwrite it to zero during major compaction. 
> KVs before major compaction will remain using system timestamp. And this 
> should be configurable, so that we should not mess up if the hbase timestamp 
> is specified by application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5674) add support in HBase to overwrite hbase timestamp to a version number during major compaction

[He Yongqiang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242094#comment-13242094
 ] 

He Yongqiang commented on HBASE-5674:
-

I use the term 'researchy' as it is mentioned so in one email thread. refer to 
http://osdir.com/ml/general/2012-03/msg52707.html I have no idea how this term 
come up.

bq. The most of us working on hbase are trying to make it an hardcore 
production worthy platform. 'Pluggable' and 'research', at least on first 
blush, sound like distractions from the project objective.
So are you referring this as conflicting with your 'hardcore production worthy 
platform' goal? 

> add support in HBase to overwrite hbase timestamp to a version number during 
> major compaction
> -
>
> Key: HBASE-5674
> URL: https://issues.apache.org/jira/browse/HBASE-5674
> Project: HBase
>  Issue Type: Improvement
>Reporter: He Yongqiang
>Assignee: He Yongqiang
>
> Right now, a millisecond-level timestamp is attached to every record. 
> In our case, we only need a version number (mostly it will be just zero etc). 
> A millisecond timestamp is too heavy to carry. We should add support to 
> overwrite it to zero during major compaction. 
> KVs before major compaction will remain using system timestamp. And this 
> should be configurable, so that we should not mess up if the hbase timestamp 
> is specified by application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5535) Make the functions in task monitor synchronized

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5535:
-

Fix Version/s: 0.92.2

Committed to 0.92 branch at Jon's prompting (thanks for the reminder)

> Make the functions in task monitor synchronized
> ---
>
> Key: HBASE-5535
> URL: https://issues.apache.org/jira/browse/HBASE-5535
> Project: HBase
>  Issue Type: Bug
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Fix For: 0.92.2, 0.94.0
>
> Attachments: 
> HBASE-5535-Make-the-functions-in-task-monitor-synchr-2012-03-08_16_33_42.patch
>
>
> There are some potential race condition in the task monitor. So update the 
> functions in task monitor to be synchronized.
> The example of the problem caused by the race condition:
> ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush 
> failed for region 
> java.lang.IndexOutOfBoundsException: Index: 1745, Size: 1744
> at java.util.ArrayList.add(ArrayList.java:367)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.AbstractList.add(AbstractList.java:91)
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:74)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1139)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:260)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:234)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5674) add support in HBase to overwrite hbase timestamp to a version number during major compaction

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242085#comment-13242085
 ] 

stack commented on HBASE-5674:
--

bq. I hope this can be done in open source hbase, and can be pluggable.

Can you do your research w/o requiring that your 'researchy' code be committed 
to core.  The most of us working on hbase are trying to make it an hardcore 
production worthy platform.  'Pluggable' and 'research', at least on first 
blush, sound like distractions from the project objective.

But maybe the research aligns with where hbase is trying to go.  Whats your 
research on?

Thanks.

> add support in HBase to overwrite hbase timestamp to a version number during 
> major compaction
> -
>
> Key: HBASE-5674
> URL: https://issues.apache.org/jira/browse/HBASE-5674
> Project: HBase
>  Issue Type: Improvement
>Reporter: He Yongqiang
>Assignee: He Yongqiang
>
> Right now, a millisecond-level timestamp is attached to every record. 
> In our case, we only need a version number (mostly it will be just zero etc). 
> A millisecond timestamp is too heavy to carry. We should add support to 
> overwrite it to zero during major compaction. 
> KVs before major compaction will remain using system timestamp. And this 
> should be configurable, so that we should not mess up if the hbase timestamp 
> is specified by application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1697) Discretionary access control

[Laxman (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242081#comment-13242081
 ] 

Laxman commented on HBASE-1697:
---

Thanks Gary for the info on Security.
I'm going through the current implementation.
Soon will take up some jiras.

> Discretionary access control
> 
>
> Key: HBASE-1697
> URL: https://issues.apache.org/jira/browse/HBASE-1697
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>
> Consider implementing discretionary access control for HBase.
> Access control has three aspects: authentication, authorization and audit.
> - Authentication: Access is controlled by insisting on an authentication 
> procedure to establish the identity of the user. The authentication procedure 
> should minimally require a non-plaintext authentication factor (e.g. 
> encrypted password with salt) and should ideally or at least optionally 
> provide cryptographically strong confidence via public key certification.
> - Authorization: Access is controlled by specifying rights to resources via 
> an access control list (ACL). An ACL is a list of permissions attached to an 
> object. The list specifies who or what is allowed to access the object and 
> what operations are allowed to be performed on the object, f.e. create, 
> update, read, or delete.
> - Audit: Important actions taken by subjects should be logged for 
> accountability, a chronological record which  enables the full reconstruction 
> and examination of a sequence of events, e.g. schema changes or data 
> mutations. Logging activity should be protected from all subjects except for 
> a restricted set with administrative privilege, perhaps to only a single 
> super-user. 
> Discretionary access control means the access policy for an object is 
> determined by the owner of the object. Every object in the system must have a 
> valid owner. Owners can assign access rights and permissions to other users. 
> The initial owner of an object is the subject who created it. If subjects are 
> deleted from a system, ownership of objects owned by them should revert to 
> some super-user or otherwise valid default. 
> HBase can enforce access policy at table, column family, or cell granularity. 
> Cell granularity does not make much sense. An implementation which controls 
> access at both the table and column family levels is recommended, though a 
> first cut could consider control at the table level only. The initial set of 
> permissions can be: Create (table schema or column family), update (table 
> schema or column family), read (column family), delete (table or column 
> family), execute (filters), and transfer ownership. The subject identities 
> and access tokens could be stored in a new administrative table. ACLs on 
> tables and column families can be stored in META. 
> Access other than read access to catalog and administrative tables should be 
> restricted to a set of administrative users or perhaps a single super-user. A 
> data mutation on a user table by a subject without administrative or 
> superuser privilege which results in a table split is an implicit temporary 
> privilege elevation where the regionserver or master updates the catalog 
> tables as necessary to support the split. 
> Audit logging should be configurable on a per-table basis to avoid this 
> overhead where it is not wanted.
> Consider supporting external authentication and subject identification 
> mechanisms with Java library support: RADIUS/TACACS, Kerberos, LDAP.
> Consider logging audit trails to an HBase table (bigtable type schemas are 
> natural for this) and optionally external logging options with Java library 
> support -- syslog, etc., or maybe commons-logging is sufficient and punt to 
> administrator to set up appropriate commons-logging/log4j configurations for 
> their needs. 
> If HBASE-1002 is considered, and the option to support filtering via upload 
> of (perhaps complex) bytecode produced by some little language compiler is 
> implemented, the execute privilege could be extended in a manner similar to 
> how stored procedures in SQL land execute either with the privilege of the 
> current user or the (table/procedure) creator.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5673) The OOM problem of IPC client call cause all handle block

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242079#comment-13242079
 ] 

stack commented on HBASE-5673:
--

No problem Xufeng.

I reverted the v1 patch from all places and then applied everywhere your v2 
patch.

> The OOM problem of IPC client call  cause all handle block
> --
>
> Key: HBASE-5673
> URL: https://issues.apache.org/jira/browse/HBASE-5673
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
> Environment: 0.90.6
>Reporter: xufeng
>Assignee: xufeng
> Fix For: 0.90.7, 0.92.2, 0.94.1
>
> Attachments: HBASE-5673-90-V2.patch, HBASE-5673-90.patch
>
>
> if HBaseClient meet "unable to create new native thread" exception, the call 
> will never complete because it be lost in calls queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5604) HLog replay tool that generates HFiles for use by LoadIncrementalHFiles.

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242076#comment-13242076
 ] 

stack commented on HBASE-5604:
--

I like the name WALPlayer.

How is this related to Import at all?  (Import imports WAL files?)

Does Import write HFiles?

Does the hdfs mod time get updated when you close the files?  If so, that might 
work for when the file is in postion but won't work if file gets moved to 
.oldlogs or to archive?  Should we add a tail on a WAL with metadata of fixed 
size so you know where to start reading to pick it up?  I suppose you can't 
rely on the fact that all WALs are present?  If they were, you could use the 
start date of the next WAL (after sorting them by date) as the ending date of 
the current file.

Should we rename WALs on close so they have the start and end time as their 
name?



> HLog replay tool that generates HFiles for use by LoadIncrementalHFiles.
> 
>
> Key: HBASE-5604
> URL: https://issues.apache.org/jira/browse/HBASE-5604
> Project: HBase
>  Issue Type: New Feature
>Reporter: Lars Hofhansl
>
> Just an idea I had. Might be useful for restore of a backup using the HLogs.
> This could an M/R (with a mapper per HLog file).
> The tool would get a timerange and a (set of) table(s). We'd pick the right 
> HLogs based on time before the M/R job is started and then have a mapper per 
> HLog file.
> The mapper would then go through the HLog, filter all WALEdits that didn't 
> fit into the time range or are not any of the tables and then uses 
> HFileOutputFormat to generate HFiles.
> Would need to indicate the splits we want, probably from a live table.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4348) Add metrics for regions in transition

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242074#comment-13242074
 ] 

jirapos...@reviews.apache.org commented on HBASE-4348:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4402/
---

(Updated 2012-03-30 05:21:12.616277)


Review request for hbase.


Changes
---

Changes done as per Jon and Ted's comments. Thanks.


Summary
---

This patch is for adding Region in transition metrics to the HMaster metrics 
system. It also adds these metrics in the master ui, in the Region in 
transition section. I have attached the proposed new format in the jira 4348.


This addresses bug HBase-4348.
https://issues.apache.org/jira/browse/HBase-4348


Diffs (updated)
-

  
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon
 0dc0691 
  src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 64def15 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace 
  src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 
83abc52 
  src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
91dce36 

Diff: https://reviews.apache.org/r/4402/diff


Testing
---

Ran on a 5 node cluster and kill region servers randomly to observe the changes 
in the RIT metrics as emitted out by the Master's mxbean;

mvn test passes without any failure.


Thanks,

Himanshu



> Add metrics for regions in transition
> -
>
> Key: HBASE-4348
> URL: https://issues.apache.org/jira/browse/HBASE-4348
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Himanshu Vashishtha
>Priority: Minor
>  Labels: noob
> Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, 
> RITs.png, RegionInTransitions2.png, metrics-v2.patch
>
>
> The following metrics would be useful for monitoring the master:
> - the number of regions in transition
> - the number of regions in transition that have been in transition for more 
> than a minute
> - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5673) The OOM problem of IPC client call cause all handle block

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242073#comment-13242073
 ] 

Hudson commented on HBASE-5673:
---

Integrated in HBase-0.94 #68 (See 
[https://builds.apache.org/job/HBase-0.94/68/])
HBASE-5673 The OOM problem of IPC client call cause all handle block 
(Revision 1307242)

 Result = FAILURE
stack : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java


> The OOM problem of IPC client call  cause all handle block
> --
>
> Key: HBASE-5673
> URL: https://issues.apache.org/jira/browse/HBASE-5673
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
> Environment: 0.90.6
>Reporter: xufeng
>Assignee: xufeng
> Fix For: 0.90.7, 0.92.2, 0.94.1
>
> Attachments: HBASE-5673-90-V2.patch, HBASE-5673-90.patch
>
>
> if HBaseClient meet "unable to create new native thread" exception, the call 
> will never complete because it be lost in calls queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5673) The OOM problem of IPC client call cause all handle block

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242072#comment-13242072
 ] 

Hudson commented on HBASE-5673:
---

Integrated in HBase-0.92 #345 (See 
[https://builds.apache.org/job/HBase-0.92/345/])
HBASE-5673 The OOM problem of IPC client call cause all handle block 
(Revision 1307246)

 Result = FAILURE
stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java


> The OOM problem of IPC client call  cause all handle block
> --
>
> Key: HBASE-5673
> URL: https://issues.apache.org/jira/browse/HBASE-5673
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
> Environment: 0.90.6
>Reporter: xufeng
>Assignee: xufeng
> Fix For: 0.90.7, 0.92.2, 0.94.1
>
> Attachments: HBASE-5673-90-V2.patch, HBASE-5673-90.patch
>
>
> if HBaseClient meet "unable to create new native thread" exception, the call 
> will never complete because it be lost in calls queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5619) Create PB protocols for HRegionInterface

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5619:
-

Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

Let me try it against hadoopqa

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: 5619v6.txt, hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch, hbase-5619_v5.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5619) Create PB protocols for HRegionInterface

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5619:
-

Attachment: 5619v6.txt

This is Jimmy's patch with a package.html added at o.a.h.h.protobuf to say 
whats going on in the package and then a README at src/main/protobuf with 
directions and a bit of script (from pom.xml amended) on how to regen the 
classes if you make changes to definition files.

I tried to find a way to add an apache license to the generated files -- its 
probably not necessary... they have headers which say they are generated files 
-- and a bit more doc on where the proto files are, etc., but I couldn't figure 
how from reading protobuf documentation.

Also excluded new generated files from findbugs report

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: 5619v6.txt, hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch, hbase-5619_v5.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5619) Create PB protocols for HRegionInterface

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5619:
-

Status: Open  (was: Patch Available)

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: 5619v6.txt, hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch, hbase-5619_v5.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5604) HLog replay tool that generates HFiles for use by LoadIncrementalHFiles.

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242067#comment-13242067
 ] 

Lars Hofhansl commented on HBASE-5604:
--

I have something basic working now. Needs a bunch of polishing, but it works in 
principle.

Right now I have this hooked up with Import. I.e. Import can optionally import 
from a directory that contains HLog files (at any depth).

Does it make sense to keep this with Import? The advantage is that there is one 
place for Importing stuff, and things like CF mapping exist already. On the 
other hand Import is becoming a bit overloaded with options now and something 
PlayLogs might be better along ImportTsv and Import. Can still work out how to 
share some of the code.

Also from the name of an HLog file I can tell when the first entry was written 
to it, but not what the last entry is. Is there a way to find this out? It 
would allow me to filter HLog files of it is known that they do not fall in the 
requested time range.


> HLog replay tool that generates HFiles for use by LoadIncrementalHFiles.
> 
>
> Key: HBASE-5604
> URL: https://issues.apache.org/jira/browse/HBASE-5604
> Project: HBase
>  Issue Type: New Feature
>Reporter: Lars Hofhansl
>
> Just an idea I had. Might be useful for restore of a backup using the HLogs.
> This could an M/R (with a mapper per HLog file).
> The tool would get a timerange and a (set of) table(s). We'd pick the right 
> HLogs based on time before the M/R job is started and then have a mapper per 
> HLog file.
> The mapper would then go through the HLog, filter all WALEdits that didn't 
> fit into the time range or are not any of the tables and then uses 
> HFileOutputFormat to generate HFiles.
> Would need to indicate the splits we want, probably from a live table.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242065#comment-13242065
 ] 

Hudson commented on HBASE-5564:
---

Integrated in HBase-TRUNK-security #154 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/154/])
HBASE-5564 Bulkload is discarding duplicate records (Revision 1306907)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java


> Bulkload is discarding duplicate records
> 
>
> Key: HBASE-5564
> URL: https://issues.apache.org/jira/browse/HBASE-5564
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.96.0
> Environment: HBase 0.92
>Reporter: Laxman
>Assignee: Laxman
>  Labels: bulkloader
> Fix For: 0.96.0
>
> Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, 
> HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, 
> HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch
>
>
> Duplicate records are getting discarded when duplicate records exists in same 
> input file and more specifically if they exists in same split.
> Duplicate records are considered if the records are from diffrent different 
> splits.
> Version under test: HBase 0.92

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242066#comment-13242066
 ] 

Hudson commented on HBASE-5097:
---

Integrated in HBase-TRUNK-security #154 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/154/])
HBASE-5097 RegionObserver implementation whose preScannerOpen and 
postScannerOpen Impl return null can stall the system initialization through 
NPE (Ram) (Revision 1307036)

 Result = FAILURE
ramkrishna : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


> RegionObserver implementation whose preScannerOpen and postScannerOpen Impl 
> return null can stall the system initialization through NPE
> ---
>
> Key: HBASE-5097
> URL: https://issues.apache.org/jira/browse/HBASE-5097
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5097.patch, HBASE-5097_1.patch, HBASE-5097_2.patch
>
>
> In HRegionServer.java openScanner()
> {code}
>   r.prepareScanner(scan);
>   RegionScanner s = null;
>   if (r.getCoprocessorHost() != null) {
> s = r.getCoprocessorHost().preScannerOpen(scan);
>   }
>   if (s == null) {
> s = r.getScanner(scan);
>   }
>   if (r.getCoprocessorHost() != null) {
> s = r.getCoprocessorHost().postScannerOpen(scan, s);
>   }
> {code}
> If we dont have implemention for postScannerOpen the RegionScanner is null 
> and so throwing nullpointer 
> {code}
> java.lang.NullPointerException
>   at 
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
> {code}
> Making this defect as blocker.. Pls feel free to change the priority if am 
> wrong.  Also correct me if my way of trying out coprocessors without 
> implementing postScannerOpen is wrong.  Am just a learner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5673) The OOM problem of IPC client call cause all handle block

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242064#comment-13242064
 ] 

Hudson commented on HBASE-5673:
---

Integrated in HBase-TRUNK-security #154 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/154/])
HBASE-5673 The OOM problem of IPC client call cause all handle block 
(Revision 1307240)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java


> The OOM problem of IPC client call  cause all handle block
> --
>
> Key: HBASE-5673
> URL: https://issues.apache.org/jira/browse/HBASE-5673
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
> Environment: 0.90.6
>Reporter: xufeng
>Assignee: xufeng
> Fix For: 0.90.7, 0.92.2, 0.94.1
>
> Attachments: HBASE-5673-90-V2.patch, HBASE-5673-90.patch
>
>
> if HBaseClient meet "unable to create new native thread" exception, the call 
> will never complete because it be lost in calls queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5633) NPE reading ZK config in HBase

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242060#comment-13242060
 ] 

Hudson commented on HBASE-5633:
---

Integrated in HBase-TRUNK-security #154 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/154/])
HBASE-5638 Readability improvements on HBASE-5633: NPE reading ZK config in 
HBase (Matteo Bertozzi) (Revision 1307085)

 Result = FAILURE
jmhsieh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java


> NPE reading ZK config in HBase
> --
>
> Key: HBASE-5633
> URL: https://issues.apache.org/jira/browse/HBASE-5633
> Project: HBase
>  Issue Type: Bug
>  Components: zookeeper
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
> HBASE-5633-v1.patch, HBASE-5633-v2.patch
>
>
> If zoo.cfg contains server.* ("server.0=server0:2888:3888\n") and 
> cluster.distributed property (in hbase-site.xml) is empty we get an NPE in 
> parseZooCfg().
> The easy way to reproduce the bug is running 
> org.apache.hbase.zookeeper.TestHQuorumPeer with hbase-site.xml containing:
> {code}
> 
>   hbase.cluster.distributed
>   
> 
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5670) Have Mutation implement the Row interface.

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242061#comment-13242061
 ] 

Hudson commented on HBASE-5670:
---

Integrated in HBase-TRUNK-security #154 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/154/])
HBASE-5670 Have Mutation implement the Row interface. (Revision 1306959)

 Result = FAILURE
larsh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Append.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java


> Have Mutation implement the Row interface.
> --
>
> Key: HBASE-5670
> URL: https://issues.apache.org/jira/browse/HBASE-5670
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Trivial
> Fix For: 0.96.0, 0.94.1
>
> Attachments: 5670-0.94.txt, 5670-trunk.txt, 5670-trunk.txt
>
>
> In HBASE-4347 I factored some code from Put/Delete/Append in Mutation.
> In a discussion with a co-worker I noticed that Put/Delete/Append still 
> implement the Row interface, but Mutation does not.
> In a trivial change I would like to move that interface up to Mutation, along 
> with changing HTable.batch(List) to HTable.batch(List) 
> (HConnection.processBatch takes List already anyway), so that 
> HTable.batch can be used with a list of Mutations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242063#comment-13242063
 ] 

Hudson commented on HBASE-5638:
---

Integrated in HBase-TRUNK-security #154 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/154/])
HBASE-5638 Readability improvements on HBASE-5633: NPE reading ZK config in 
HBase (Matteo Bertozzi) (Revision 1307085)

 Result = FAILURE
jmhsieh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java


> Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
> --
>
> Key: HBASE-5638
> URL: https://issues.apache.org/jira/browse/HBASE-5638
> Project: HBase
>  Issue Type: Sub-task
>  Components: zookeeper
>Affects Versions: 0.90.6, 0.92.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.94.1
>
> Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
> HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, 
> HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4398) If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml.

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242062#comment-13242062
 ] 

Hudson commented on HBASE-4398:
---

Integrated in HBase-TRUNK-security #154 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/154/])
HBASE-4398 If HRegionPartitioner is used in MapReduce, client side 
configurations are overwritten by hbase-site.xml (Revision 1307116)

 Result = FAILURE
stack : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java


> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> -
>
> Key: HBASE-4398
> URL: https://issues.apache.org/jira/browse/HBASE-4398
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.90.4
>Reporter: Takuya Ueshin
>Assignee: Takuya Ueshin
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-4398.patch
>
>
> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> We can reproduce the problem by the following instructions:
> {noformat}
> - Add HRegionPartitioner.class to the 4th argument of
> TableMapReduceUtil#initTableReducerJob()
> at line around 133
> in src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
> - Change or remove "hbase.zookeeper.property.clientPort" property
> in hbase-site.xml ( for example, changed to 12345 ).
> - run testMultiRegionTable()
> {noformat}
> Then I got error messages as following:
> {noformat}
> 2011-09-12 22:28:51,020 DEBUG [Thread-832] zookeeper.ZKUtil(93): hconnection 
> opening connection to ZooKeeper with ensemble (localhost:12345)
> 2011-09-12 22:28:51,022 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(89): The identifier of this process is 
> 43200@imac.local
> 2011-09-12 22:28:51,123 WARN  [Thread-832] 
> zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: 
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for /hbase/master
> 2011-09-12 22:28:51,123 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after 
> sleeping 1000 ms
>  =
> 2011-09-12 22:29:02,418 ERROR [Thread-832] mapreduce.HRegionPartitioner(125): 
> java.io.IOException: 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2e54e48d
>  closed
> 2011-09-12 22:29:02,422 WARN  [Thread-832] mapred.LocalJobRunner$Job(256): 
> job_local_0001
> java.lang.NullPointerException
>at 
> org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:128)
>at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:527)
>at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> {noformat}
> I think HTable should connect to ZooKeeper at port 21818 configured at client 
> side instead of 12345 in hbase-site.xml
> and It might be caused by "HBaseConfiguration.addHbaseResources(conf);" in 
> HRegionPartitioner#setConf(Configuration).
> And this might mean that all of client side configurations, also configured 
> in hbase-site.xml, are overwritten caused by this problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5673) The OOM problem of IPC client call cause all handle block

[xufeng (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xufeng updated HBASE-5673:
--

Attachment: HBASE-5673-90-V2.patch

@Stack
Sorry,I attach a wrong patch.
I updated it.name:HBASE-5673-90-V2.patch

> The OOM problem of IPC client call  cause all handle block
> --
>
> Key: HBASE-5673
> URL: https://issues.apache.org/jira/browse/HBASE-5673
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
> Environment: 0.90.6
>Reporter: xufeng
>Assignee: xufeng
> Fix For: 0.90.7, 0.92.2, 0.94.1
>
> Attachments: HBASE-5673-90-V2.patch, HBASE-5673-90.patch
>
>
> if HBaseClient meet "unable to create new native thread" exception, the call 
> will never complete because it be lost in calls queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-5673) The OOM problem of IPC client call cause all handle block

[stack (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-5673.
--

   Resolution: Fixed
Fix Version/s: 0.94.1
   0.92.2
   0.90.7
 Hadoop Flags: Reviewed

Committed to 0.90, 0.92, 0.94 and trunk.  Thanks for the patch Xufeng

> The OOM problem of IPC client call  cause all handle block
> --
>
> Key: HBASE-5673
> URL: https://issues.apache.org/jira/browse/HBASE-5673
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
> Environment: 0.90.6
>Reporter: xufeng
>Assignee: xufeng
> Fix For: 0.90.7, 0.92.2, 0.94.1
>
> Attachments: HBASE-5673-90.patch
>
>
> if HBaseClient meet "unable to create new native thread" exception, the call 
> will never complete because it be lost in calls queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row

[Anoop Sam John (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242048#comment-13242048
 ] 

Anoop Sam John commented on HBASE-5664:
---

Previosly there was no use to give CP hook as there were no seek available in 
RegionScanner. Now with HBASE-5520 reseek() is provided in RegionScanner, which 
the CP hook can use for fast forward.

> CP hooks in Scan flow for fast forward when filter filters out a row
> 
>
> Key: HBASE-5664
> URL: https://issues.apache.org/jira/browse/HBASE-5664
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Affects Versions: 0.92.1
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.96.0
>
>
> In HRegion.nextInternal(int limit, String metric)
>   We have while(true) loop so as to fetch a next result which satisfies 
> filter condition. When Filter filters out the current fetched row we call 
> nextRow(byte [] currentRow) before going with the next row.
> {code}
> if (results.isEmpty() || filterRow()) {
> // this seems like a redundant step - we already consumed the row
> // there're no left overs.
> // the reasons for calling this method are:
> // 1. reset the filters.
> // 2. provide a hook to fast forward the row (used by subclasses)
> nextRow(currentRow);
> {code}
> // 2. provide a hook to fast forward the row (used by subclasses)
> We can provide same feature of fast forward support for the CP also.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row

[Anoop Sam John (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242047#comment-13242047
 ] 

Anoop Sam John commented on HBASE-5664:
---

Previously seeking was not possible on the RegionScanner. Now with HBASE-5520 
CP can do a fast forward with reseek() on RegionScanner.
Use case of this new hook


> CP hooks in Scan flow for fast forward when filter filters out a row
> 
>
> Key: HBASE-5664
> URL: https://issues.apache.org/jira/browse/HBASE-5664
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Affects Versions: 0.92.1
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.96.0
>
>
> In HRegion.nextInternal(int limit, String metric)
>   We have while(true) loop so as to fetch a next result which satisfies 
> filter condition. When Filter filters out the current fetched row we call 
> nextRow(byte [] currentRow) before going with the next row.
> {code}
> if (results.isEmpty() || filterRow()) {
> // this seems like a redundant step - we already consumed the row
> // there're no left overs.
> // the reasons for calling this method are:
> // 1. reset the filters.
> // 2. provide a hook to fast forward the row (used by subclasses)
> nextRow(currentRow);
> {code}
> // 2. provide a hook to fast forward the row (used by subclasses)
> We can provide same feature of fast forward support for the CP also.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5673) The OOM problem of IPC client call cause all handle block

[xufeng (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xufeng updated HBASE-5673:
--

Attachment: HBASE-5673-90.patch

this patch for 0.90
unit test is running now.

I have checked the code of trunk and 0.92 and also have this issue.

pls review the 90 patch and give me some suggestions,thanks.

> The OOM problem of IPC client call  cause all handle block
> --
>
> Key: HBASE-5673
> URL: https://issues.apache.org/jira/browse/HBASE-5673
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
> Environment: 0.90.6
>Reporter: xufeng
>Assignee: xufeng
> Attachments: HBASE-5673-90.patch
>
>
> if HBaseClient meet "unable to create new native thread" exception, the call 
> will never complete because it be lost in calls queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242040#comment-13242040
 ] 

Hadoop QA commented on HBASE-5619:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12520550/hbase-5619_v5.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 68 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestImportTsv

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1350//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1350//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1350//console

This message is automatically generated.

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch, hbase-5619_v5.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_

[fulin wang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242035#comment-13242035
 ] 

fulin wang commented on HBASE-5599:
---

Thanks Jonathan Hsieh for review.
1. the NO_VERSION_FILE, I will add '-fixVersionFile' to fix this error and test 
case.
2. the SHOULD_NOT_BE_DEPLOYED fault scensrios, I change disable code that it is 
not call bulkAssign method, so the region is deployed but the table is 
disable.I do not know how to write this test case.

> The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, 
> NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, 
> FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN.
> 
>
> Key: HBASE-5599
> URL: https://issues.apache.org/jira/browse/HBASE-5599
> Project: HBase
>  Issue Type: New Feature
>  Components: hbck
>Affects Versions: 0.90.6
>Reporter: fulin wang
> Fix For: 0.90.6
>
> Attachments: hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, 
> hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, hbase-5599-0.92_v5.patch, 
> hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch
>
>
> The hbck tool can not fix the six scenarios.
> 1. Version file does not exist in root dir.
>Fix: I try to create a version file by 'FSUtils.setVersion' method.
>
> 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any 
> region server.
>Fix: I get region info form the hdfs file, this region info write to 
> '.META.' table.
>
> 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME]
>Fix: I get region info form the hdfs file, this region info write to 
> '.META.' table.
>
> 4. [REGIONNAME] should not be deployed according to META, but is deployed on 
> [SERVERNAME]
>Fix: Close this region.
>
> 5. First region should start with an empty key.  You need to  create a new 
> region and regioninfo in HDFS to plug the hole.
>Fix: The region info is not in hdfs and .META., so it create a empty 
> region for this error.
> 6. There is a hole in the region chain between [KEY] and [KEY]. You need to 
> create a new regioninfo and region dir in hdfs to plug the hole.
>   Fix: The region info is not in hdfs and .META., so it create a empty region 
> for this hole.
>   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5674) add support in HBase to overwrite hbase timestamp to a version number during major compaction

[He Yongqiang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242028#comment-13242028
 ] 

He Yongqiang commented on HBASE-5674:
-

bq. For whom?

For our 'researchy' project...

bq. Can you not just have your client specify timestamp of 0?

I hope this can be done in open source hbase, and can be pluggable. 

> add support in HBase to overwrite hbase timestamp to a version number during 
> major compaction
> -
>
> Key: HBASE-5674
> URL: https://issues.apache.org/jira/browse/HBASE-5674
> Project: HBase
>  Issue Type: Improvement
>Reporter: He Yongqiang
>Assignee: He Yongqiang
>
> Right now, a millisecond-level timestamp is attached to every record. 
> In our case, we only need a version number (mostly it will be just zero etc). 
> A millisecond timestamp is too heavy to carry. We should add support to 
> overwrite it to zero during major compaction. 
> KVs before major compaction will remain using system timestamp. And this 
> should be configurable, so that we should not mess up if the hbase timestamp 
> is specified by application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5619) Create PB protocols for HRegionInterface

[Jimmy Xiang (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5619:
---

Status: Patch Available  (was: Open)

I reverted the pom file change, and checked in the generated protobuf classes.

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch, hbase-5619_v5.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5619) Create PB protocols for HRegionInterface

[Jimmy Xiang (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5619:
---

Status: Open  (was: Patch Available)

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch, hbase-5619_v5.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5619) Create PB protocols for HRegionInterface

[Jimmy Xiang (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5619:
---

Attachment: hbase-5619_v5.patch

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch, hbase-5619_v5.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5655) Cap space usage of default log4j rolling policy

[Zhihong Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5655:
--

Fix Version/s: 0.96.0
 Hadoop Flags: Reviewed

Will integrate tomorrow if there is no objection.

> Cap space usage of default log4j rolling policy
> ---
>
> Key: HBASE-5655
> URL: https://issues.apache.org/jira/browse/HBASE-5655
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.96.0
>
> Attachments: 5655-v1.patch, HBase-5655-v2.patch, HBase-5655-v3.patch
>
>
> The current default log4j policy is to use Daily Rolling File Appender 
> (DRFA). At times, its good to have a cap on the maximum size of the logs in 
> order to limit its disk usage. Here is a proposal to set a new file appemder 
> (RFA) as the default appender. It can be configured via env so that existing 
> tools can use the current behavior of using DRFA instead. 
> This is in parallel with jira Hadoop-8149.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1697) Discretionary access control

[Gary Helmling (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241997#comment-13241997
 ] 

Gary Helmling commented on HBASE-1697:
--

@Laxman,

We probably need to do some JIRA housekeeping here.  There hasn't been much 
activity in this ticket because it's really just an umbrella issue tying 
together the individual efforts.

Initial security features are in place with HBASE-3025, HBASE-2418, and 
HBASE-2742.  These are released in 0.92.  Currently access control is performed 
by use of ACLs stored at the global (forthcoming), table, column family, or 
column qualifier level.

There are definitely additional features that can be built to contribute to our 
security solution:
* adding client authentication for thrift and REST clients
* proxying HBase access from thrift and REST servers as the authenticated 
clients (currently these server can simply be configured to authenticate and 
access HBase as their own principals)
* supporting or moving to alternate access control schemes (RBAC)
* row or key-value based access control
* supporting other authentication mechanisms than kerberos and authentication 
tokens
* probably many others

Anyone interested in working on these would certainly be welcomed.  But we do 
currently have a working security implementation that integrates nicely with 
HDFS and map reduce.

> Discretionary access control
> 
>
> Key: HBASE-1697
> URL: https://issues.apache.org/jira/browse/HBASE-1697
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>
> Consider implementing discretionary access control for HBase.
> Access control has three aspects: authentication, authorization and audit.
> - Authentication: Access is controlled by insisting on an authentication 
> procedure to establish the identity of the user. The authentication procedure 
> should minimally require a non-plaintext authentication factor (e.g. 
> encrypted password with salt) and should ideally or at least optionally 
> provide cryptographically strong confidence via public key certification.
> - Authorization: Access is controlled by specifying rights to resources via 
> an access control list (ACL). An ACL is a list of permissions attached to an 
> object. The list specifies who or what is allowed to access the object and 
> what operations are allowed to be performed on the object, f.e. create, 
> update, read, or delete.
> - Audit: Important actions taken by subjects should be logged for 
> accountability, a chronological record which  enables the full reconstruction 
> and examination of a sequence of events, e.g. schema changes or data 
> mutations. Logging activity should be protected from all subjects except for 
> a restricted set with administrative privilege, perhaps to only a single 
> super-user. 
> Discretionary access control means the access policy for an object is 
> determined by the owner of the object. Every object in the system must have a 
> valid owner. Owners can assign access rights and permissions to other users. 
> The initial owner of an object is the subject who created it. If subjects are 
> deleted from a system, ownership of objects owned by them should revert to 
> some super-user or otherwise valid default. 
> HBase can enforce access policy at table, column family, or cell granularity. 
> Cell granularity does not make much sense. An implementation which controls 
> access at both the table and column family levels is recommended, though a 
> first cut could consider control at the table level only. The initial set of 
> permissions can be: Create (table schema or column family), update (table 
> schema or column family), read (column family), delete (table or column 
> family), execute (filters), and transfer ownership. The subject identities 
> and access tokens could be stored in a new administrative table. ACLs on 
> tables and column families can be stored in META. 
> Access other than read access to catalog and administrative tables should be 
> restricted to a set of administrative users or perhaps a single super-user. A 
> data mutation on a user table by a subject without administrative or 
> superuser privilege which results in a table split is an implicit temporary 
> privilege elevation where the regionserver or master updates the catalog 
> tables as necessary to support the split. 
> Audit logging should be configurable on a per-table basis to avoid this 
> overhead where it is not wanted.
> Consider supporting external authentication and subject identification 
> mechanisms with Java library support: RADIUS/TACACS, Kerberos, LDAP.
> Consider logging audit trails to an HBase table (bigtable type schemas are 
> natural for this) and optionally external logging o

[jira] [Commented] (HBASE-4398) If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml.

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241994#comment-13241994
 ] 

Hudson commented on HBASE-4398:
---

Integrated in HBase-0.92 #344 (See 
[https://builds.apache.org/job/HBase-0.92/344/])
HBASE-4398 If HRegionPartitioner is used in MapReduce, client side 
configurations are overwritten by hbase-site.xml (Revision 1307118)

 Result = SUCCESS
stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java


> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> -
>
> Key: HBASE-4398
> URL: https://issues.apache.org/jira/browse/HBASE-4398
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.90.4
>Reporter: Takuya Ueshin
>Assignee: Takuya Ueshin
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-4398.patch
>
>
> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> We can reproduce the problem by the following instructions:
> {noformat}
> - Add HRegionPartitioner.class to the 4th argument of
> TableMapReduceUtil#initTableReducerJob()
> at line around 133
> in src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
> - Change or remove "hbase.zookeeper.property.clientPort" property
> in hbase-site.xml ( for example, changed to 12345 ).
> - run testMultiRegionTable()
> {noformat}
> Then I got error messages as following:
> {noformat}
> 2011-09-12 22:28:51,020 DEBUG [Thread-832] zookeeper.ZKUtil(93): hconnection 
> opening connection to ZooKeeper with ensemble (localhost:12345)
> 2011-09-12 22:28:51,022 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(89): The identifier of this process is 
> 43200@imac.local
> 2011-09-12 22:28:51,123 WARN  [Thread-832] 
> zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: 
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for /hbase/master
> 2011-09-12 22:28:51,123 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after 
> sleeping 1000 ms
>  =
> 2011-09-12 22:29:02,418 ERROR [Thread-832] mapreduce.HRegionPartitioner(125): 
> java.io.IOException: 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2e54e48d
>  closed
> 2011-09-12 22:29:02,422 WARN  [Thread-832] mapred.LocalJobRunner$Job(256): 
> job_local_0001
> java.lang.NullPointerException
>at 
> org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:128)
>at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:527)
>at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> {noformat}
> I think HTable should connect to ZooKeeper at port 21818 configured at client 
> side instead of 12345 in hbase-site.xml
> and It might be caused by "HBaseConfiguration.addHbaseResources(conf);" in 
> HRegionPartitioner#setConf(Configuration).
> And this might mean that all of client side configurations, also configured 
> in hbase-site.xml, are overwritten caused by this problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5655) Cap space usage of default log4j rolling policy

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241989#comment-13241989
 ] 

Hadoop QA commented on HBASE-5655:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12520536/HBase-5655-v3.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestImportTsv

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1349//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1349//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1349//console

This message is automatically generated.

> Cap space usage of default log4j rolling policy
> ---
>
> Key: HBASE-5655
> URL: https://issues.apache.org/jira/browse/HBASE-5655
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Attachments: 5655-v1.patch, HBase-5655-v2.patch, HBase-5655-v3.patch
>
>
> The current default log4j policy is to use Daily Rolling File Appender 
> (DRFA). At times, its good to have a cap on the maximum size of the logs in 
> order to limit its disk usage. Here is a proposal to set a new file appemder 
> (RFA) as the default appender. It can be configured via env so that existing 
> tools can use the current behavior of using DRFA instead. 
> This is in parallel with jira Hadoop-8149.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5676) Note that dfs.support.append does not have to be enabled post 1.x

[Eli Collins (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241971#comment-13241971
 ] 

Eli Collins commented on HBASE-5676:


The "X" btw is TBD, you can follow the jira to see what version this lands in.

> Note that dfs.support.append does not have to be enabled post 1.x
> -
>
> Key: HBASE-5676
> URL: https://issues.apache.org/jira/browse/HBASE-5676
> Project: HBase
>  Issue Type: Task
>Reporter: Eli Collins
>
> In Hadoop 1.x (HADOOP-8230) we are going to enable durable sync by default, 
> and remove the dfs.support.append option. What this means for you:
> - HBase will work out of the box on Hadoop 1.x, no need to tell people to 
> re-configure
> - You no longer have to enable append (which can result in data loss) to 
> enable HBase support

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5676) Note that dfs.support.append does not have to be enabled post 1.x

[Eli Collins (Created) (JIRA)]

Note that dfs.support.append does not have to be enabled post 1.x
-

 Key: HBASE-5676
 URL: https://issues.apache.org/jira/browse/HBASE-5676
 Project: HBase
  Issue Type: Task
Reporter: Eli Collins


In Hadoop 1.x (HADOOP-8230) we are going to enable durable sync by default, and 
remove the dfs.support.append option. What this means for you:
- HBase will work out of the box on Hadoop 1.x, no need to tell people to 
re-configure
- You no longer have to enable append (which can result in data loss) to enable 
HBase support

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5655) Cap space usage of default log4j rolling policy

[Himanshu Vashishtha (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Himanshu Vashishtha updated HBASE-5655:
---

Attachment: HBase-5655-v3.patch

Thanks for the review Ted.
Corrected the grammar as suggested.


> Cap space usage of default log4j rolling policy
> ---
>
> Key: HBASE-5655
> URL: https://issues.apache.org/jira/browse/HBASE-5655
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Attachments: 5655-v1.patch, HBase-5655-v2.patch, HBase-5655-v3.patch
>
>
> The current default log4j policy is to use Daily Rolling File Appender 
> (DRFA). At times, its good to have a cap on the maximum size of the logs in 
> order to limit its disk usage. Here is a proposal to set a new file appemder 
> (RFA) as the default appender. It can be configured via env so that existing 
> tools can use the current behavior of using DRFA instead. 
> This is in parallel with jira Hadoop-8149.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241955#comment-13241955
 ] 

Jonathan Hsieh commented on HBASE-5675:
---

Mubarak, found it -- the fix is actually HBASE-4386, fixed in 0.92/0.94/trunk 
(and CDH3u3).

> Create table fails if we keep refreshing master's UI for task monitor status
> 
>
> Key: HBASE-5675
> URL: https://issues.apache.org/jira/browse/HBASE-5675
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Mubarak Seyed
>Assignee: Mubarak Seyed
>  Labels: noob
>
> I tried to create a table with 2K pre-split regions, region assignment was in 
> middle and i was keep refreshing master's web UI to find the status of the 
> task using task monitor, table creation was failed and {{META}} was showing 
> 2K regions with server location value is {{null}} and regions weren't 
> deployed onto region-servers.
> {code}
> table_ACreating table table_A
> java.io.IOException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>   at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>   at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
>   at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
>   at com.test.tools.hbase.schema.main(schema.java:627)
> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at java.util.SubList.checkForComodification(AbstractList.java:752)
>   at java.util.SubList.add(AbstractList.java:632)
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.SubList.add(AbstractList.java:633)
>   ..
>   ..
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.AbstractList.add(AbstractList.java:91)
>   at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
>   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>   at $Proxy5.createTable(Unknown Source)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
>   
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Jimmy Xiang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241954#comment-13241954
 ] 

Jimmy Xiang commented on HBASE-5619:


That's what do for thrift now, not avro.

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5667) RegexStringComparator supports java.util.regex.Pattern flags

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241940#comment-13241940
 ] 

stack commented on HBASE-5667:
--

Yeah, it looks like we can do the String plus int for flags as David suggests 
given thats all it takes to create a Pattern instance: 
http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html#compile(java.lang.String,
 int)

> RegexStringComparator supports java.util.regex.Pattern flags
> 
>
> Key: HBASE-5667
> URL: https://issues.apache.org/jira/browse/HBASE-5667
> Project: HBase
>  Issue Type: Improvement
>  Components: filters
>Reporter: David Arthur
>Priority: Minor
> Attachments: HBASE-5667.diff
>
>
> * Add constructor that takes in a Pattern
> * Add Pattern's flags to Writable fields, and actually use them when 
> recomposing the Filter

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5655) Cap space usage of default log4j rolling policy

[Zhihong Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241942#comment-13241942
 ] 

Zhihong Yu commented on HBASE-5655:
---

Patch looks good.
Minor comment:
{code}
+# In case one need to do log
{code}
The above should read 'one needs to ...'

The 2 of the test failures should have been fixed by MAPREDUCE-3583.

> Cap space usage of default log4j rolling policy
> ---
>
> Key: HBASE-5655
> URL: https://issues.apache.org/jira/browse/HBASE-5655
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Attachments: 5655-v1.patch, HBase-5655-v2.patch
>
>
> The current default log4j policy is to use Daily Rolling File Appender 
> (DRFA). At times, its good to have a cap on the maximum size of the logs in 
> order to limit its disk usage. Here is a proposal to set a new file appemder 
> (RFA) as the default appender. It can be configured via env so that existing 
> tools can use the current behavior of using DRFA instead. 
> This is in parallel with jira Hadoop-8149.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-5652) [findbugs] Fix lock release on all paths

[Gregory Chanan (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan reassigned HBASE-5652:
-

Assignee: Gregory Chanan

> [findbugs] Fix lock release on all paths 
> -
>
> Key: HBASE-5652
> URL: https://issues.apache.org/jira/browse/HBASE-5652
> Project: HBase
>  Issue Type: Sub-task
>  Components: scripts
>Reporter: Jonathan Hsieh
>Assignee: Gregory Chanan
>
> See 
> https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_MT_CORRECTNESS
> Category UL

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241938#comment-13241938
 ] 

Jonathan Hsieh commented on HBASE-5675:
---

I took a closer look and HBASE-5535 doesn't seem to really add any new 
synchronization help on the tasks list.  Another related patch is a part of 
HBASE-4057; investigating further.

> Create table fails if we keep refreshing master's UI for task monitor status
> 
>
> Key: HBASE-5675
> URL: https://issues.apache.org/jira/browse/HBASE-5675
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Mubarak Seyed
>Assignee: Mubarak Seyed
>  Labels: noob
>
> I tried to create a table with 2K pre-split regions, region assignment was in 
> middle and i was keep refreshing master's web UI to find the status of the 
> task using task monitor, table creation was failed and {{META}} was showing 
> 2K regions with server location value is {{null}} and regions weren't 
> deployed onto region-servers.
> {code}
> table_ACreating table table_A
> java.io.IOException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>   at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>   at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
>   at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
>   at com.test.tools.hbase.schema.main(schema.java:627)
> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at java.util.SubList.checkForComodification(AbstractList.java:752)
>   at java.util.SubList.add(AbstractList.java:632)
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.SubList.add(AbstractList.java:633)
>   ..
>   ..
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.AbstractList.add(AbstractList.java:91)
>   at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
>   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>   at $Proxy5.createTable(Unknown Source)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
>   
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241937#comment-13241937
 ] 

stack commented on HBASE-5619:
--

I like the idea of checking in the generated protobufs files.  They'll change 
rarely.  Let the fella who generates them have to worry about his protoc setup 
(This is what we do now for thrift and avro?)

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5633) NPE reading ZK config in HBase

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241923#comment-13241923
 ] 

Hudson commented on HBASE-5633:
---

Integrated in HBase-0.92 #343 (See 
[https://builds.apache.org/job/HBase-0.92/343/])
HBASE-5638 Readability improvements and backport HBASE-5633: NPE reading ZK 
config in HBase (Matteo Bertozzi) (Revision 1307087)

 Result = SUCCESS
jmhsieh : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java


> NPE reading ZK config in HBase
> --
>
> Key: HBASE-5633
> URL: https://issues.apache.org/jira/browse/HBASE-5633
> Project: HBase
>  Issue Type: Bug
>  Components: zookeeper
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
> HBASE-5633-v1.patch, HBASE-5633-v2.patch
>
>
> If zoo.cfg contains server.* ("server.0=server0:2888:3888\n") and 
> cluster.distributed property (in hbase-site.xml) is empty we get an NPE in 
> parseZooCfg().
> The easy way to reproduce the bug is running 
> org.apache.hbase.zookeeper.TestHQuorumPeer with hbase-site.xml containing:
> {code}
> 
>   hbase.cluster.distributed
>   
> 
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241924#comment-13241924
 ] 

Hudson commented on HBASE-5638:
---

Integrated in HBase-0.92 #343 (See 
[https://builds.apache.org/job/HBase-0.92/343/])
HBASE-5638 Readability improvements and backport HBASE-5633: NPE reading ZK 
config in HBase (Matteo Bertozzi) (Revision 1307087)

 Result = SUCCESS
jmhsieh : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java


> Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
> --
>
> Key: HBASE-5638
> URL: https://issues.apache.org/jira/browse/HBASE-5638
> Project: HBase
>  Issue Type: Sub-task
>  Components: zookeeper
>Affects Versions: 0.90.6, 0.92.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.94.1
>
> Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
> HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, 
> HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241916#comment-13241916
 ] 

Enis Soztutar commented on HBASE-5619:
--

I was suggesting adding the generated files under src/, and adding the runtime 
protobuf dependency in maven. But if it won't work, we can go with requiring 
protoc. Dynamic installation will be a little bit overkill. 

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5667) RegexStringComparator supports java.util.regex.Pattern flags

[Jimmy Xiang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241914#comment-13241914
 ] 

Jimmy Xiang commented on HBASE-5667:


Pattern is fine if we can get it.

> RegexStringComparator supports java.util.regex.Pattern flags
> 
>
> Key: HBASE-5667
> URL: https://issues.apache.org/jira/browse/HBASE-5667
> Project: HBase
>  Issue Type: Improvement
>  Components: filters
>Reporter: David Arthur
>Priority: Minor
> Attachments: HBASE-5667.diff
>
>
> * Add constructor that takes in a Pattern
> * Add Pattern's flags to Writable fields, and actually use them when 
> recomposing the Filter

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5667) RegexStringComparator supports java.util.regex.Pattern flags

[David Arthur (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241905#comment-13241905
 ] 

David Arthur commented on HBASE-5667:
-

If I'm not mistaken, a Pattern is just the pattern string and the flags. The 
constructor could just as easily be (String, int) rather than a Pattern 
instance if that jives better with pb

> RegexStringComparator supports java.util.regex.Pattern flags
> 
>
> Key: HBASE-5667
> URL: https://issues.apache.org/jira/browse/HBASE-5667
> Project: HBase
>  Issue Type: Improvement
>  Components: filters
>Reporter: David Arthur
>Priority: Minor
> Attachments: HBASE-5667.diff
>
>
> * Add constructor that takes in a Pattern
> * Add Pattern's flags to Writable fields, and actually use them when 
> recomposing the Filter

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5655) Cap space usage of default log4j rolling policy

[David S. Wang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241894#comment-13241894
 ] 

David S. Wang commented on HBASE-5655:
--

There is already a JIRA filed for this: HBASE-5608.  I highly doubt that your 
patch is causing these tests to fail.

> Cap space usage of default log4j rolling policy
> ---
>
> Key: HBASE-5655
> URL: https://issues.apache.org/jira/browse/HBASE-5655
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Attachments: 5655-v1.patch, HBase-5655-v2.patch
>
>
> The current default log4j policy is to use Daily Rolling File Appender 
> (DRFA). At times, its good to have a cap on the maximum size of the logs in 
> order to limit its disk usage. Here is a proposal to set a new file appemder 
> (RFA) as the default appender. It can be configured via env so that existing 
> tools can use the current behavior of using DRFA instead. 
> This is in parallel with jira Hadoop-8149.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4398) If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml.

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241891#comment-13241891
 ] 

Hudson commented on HBASE-4398:
---

Integrated in HBase-0.94 #67 (See 
[https://builds.apache.org/job/HBase-0.94/67/])
HBASE-4398 If HRegionPartitioner is used in MapReduce, client side 
configurations are overwritten by hbase-site.xml (Revision 1307117)

 Result = SUCCESS
stack : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java


> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> -
>
> Key: HBASE-4398
> URL: https://issues.apache.org/jira/browse/HBASE-4398
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.90.4
>Reporter: Takuya Ueshin
>Assignee: Takuya Ueshin
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-4398.patch
>
>
> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> We can reproduce the problem by the following instructions:
> {noformat}
> - Add HRegionPartitioner.class to the 4th argument of
> TableMapReduceUtil#initTableReducerJob()
> at line around 133
> in src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
> - Change or remove "hbase.zookeeper.property.clientPort" property
> in hbase-site.xml ( for example, changed to 12345 ).
> - run testMultiRegionTable()
> {noformat}
> Then I got error messages as following:
> {noformat}
> 2011-09-12 22:28:51,020 DEBUG [Thread-832] zookeeper.ZKUtil(93): hconnection 
> opening connection to ZooKeeper with ensemble (localhost:12345)
> 2011-09-12 22:28:51,022 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(89): The identifier of this process is 
> 43200@imac.local
> 2011-09-12 22:28:51,123 WARN  [Thread-832] 
> zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: 
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for /hbase/master
> 2011-09-12 22:28:51,123 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after 
> sleeping 1000 ms
>  =
> 2011-09-12 22:29:02,418 ERROR [Thread-832] mapreduce.HRegionPartitioner(125): 
> java.io.IOException: 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2e54e48d
>  closed
> 2011-09-12 22:29:02,422 WARN  [Thread-832] mapred.LocalJobRunner$Job(256): 
> job_local_0001
> java.lang.NullPointerException
>at 
> org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:128)
>at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:527)
>at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> {noformat}
> I think HTable should connect to ZooKeeper at port 21818 configured at client 
> side instead of 12345 in hbase-site.xml
> and It might be caused by "HBaseConfiguration.addHbaseResources(conf);" in 
> HRegionPartitioner#setConf(Configuration).
> And this might mean that all of client side configurations, also configured 
> in hbase-site.xml, are overwritten caused by this problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5655) Cap space usage of default log4j rolling policy

[Himanshu Vashishtha (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241883#comment-13241883
 ] 

Himanshu Vashishtha commented on HBASE-5655:


It will be good to know whether these failures are caused by this patch or are 
they just flaky.
Looking at the logs for TestTAbleMapReduce's failures, it seems metrics2 were 
not loaded, and it gave out NFE:
"attempt_20120329220930011_0001_r_02_0: 2012-03-29 22:09:50,378 WARN  
[main] impl.MetricsSystemImpl(137): Metrics system not started: Cannot locate 
configuration: tried hadoop-metrics2-reducetask.properties, 
hadoop-metrics2.properties"
Shall I look into it, or are they just flaky. I can't correlate these failures 
with log4j rolling policy change as of now.

> Cap space usage of default log4j rolling policy
> ---
>
> Key: HBASE-5655
> URL: https://issues.apache.org/jira/browse/HBASE-5655
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Attachments: 5655-v1.patch, HBase-5655-v2.patch
>
>
> The current default log4j policy is to use Daily Rolling File Appender 
> (DRFA). At times, its good to have a cap on the maximum size of the logs in 
> order to limit its disk usage. Here is a proposal to set a new file appemder 
> (RFA) as the default appender. It can be configured via env so that existing 
> tools can use the current behavior of using DRFA instead. 
> This is in parallel with jira Hadoop-8149.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Jimmy Xiang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241880#comment-13241880
 ] 

Jimmy Xiang commented on HBASE-5619:


But for proto files, other projects depend on protoc, for example HADOOP/HDFS. 
We are moving towards pb, protoc dependency should be fine.
I can try to setup a temp protoc dynamically.

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status

[Mubarak Seyed (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241877#comment-13241877
 ] 

Mubarak Seyed commented on HBASE-5675:
--

Thanks Jon.

> Create table fails if we keep refreshing master's UI for task monitor status
> 
>
> Key: HBASE-5675
> URL: https://issues.apache.org/jira/browse/HBASE-5675
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Mubarak Seyed
>Assignee: Mubarak Seyed
>  Labels: noob
>
> I tried to create a table with 2K pre-split regions, region assignment was in 
> middle and i was keep refreshing master's web UI to find the status of the 
> task using task monitor, table creation was failed and {{META}} was showing 
> 2K regions with server location value is {{null}} and regions weren't 
> deployed onto region-servers.
> {code}
> table_ACreating table table_A
> java.io.IOException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>   at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>   at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
>   at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
>   at com.test.tools.hbase.schema.main(schema.java:627)
> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at java.util.SubList.checkForComodification(AbstractList.java:752)
>   at java.util.SubList.add(AbstractList.java:632)
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.SubList.add(AbstractList.java:633)
>   ..
>   ..
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.AbstractList.add(AbstractList.java:91)
>   at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
>   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>   at $Proxy5.createTable(Unknown Source)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
>   
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5655) Cap space usage of default log4j rolling policy

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241875#comment-13241875
 ] 

Hadoop QA commented on HBASE-5655:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12520517/HBase-5655-v2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestImportTsv
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1348//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1348//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1348//console

This message is automatically generated.

> Cap space usage of default log4j rolling policy
> ---
>
> Key: HBASE-5655
> URL: https://issues.apache.org/jira/browse/HBASE-5655
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Attachments: 5655-v1.patch, HBase-5655-v2.patch
>
>
> The current default log4j policy is to use Daily Rolling File Appender 
> (DRFA). At times, its good to have a cap on the maximum size of the logs in 
> order to limit its disk usage. Here is a proposal to set a new file appemder 
> (RFA) as the default appender. It can be configured via env so that existing 
> tools can use the current behavior of using DRFA instead. 
> This is in parallel with jira Hadoop-8149.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5535) Make the functions in task monitor synchronized

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241873#comment-13241873
 ] 

Jonathan Hsieh commented on HBASE-5535:
---

Stack, looks like this hasn't made it into the 0.92.2/0.92 branch.  Shall we 
just commit from this jira and add fix version?

> Make the functions in task monitor synchronized
> ---
>
> Key: HBASE-5535
> URL: https://issues.apache.org/jira/browse/HBASE-5535
> Project: HBase
>  Issue Type: Bug
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: 
> HBASE-5535-Make-the-functions-in-task-monitor-synchr-2012-03-08_16_33_42.patch
>
>
> There are some potential race condition in the task monitor. So update the 
> functions in task monitor to be synchronized.
> The example of the problem caused by the race condition:
> ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush 
> failed for region 
> java.lang.IndexOutOfBoundsException: Index: 1745, Size: 1744
> at java.util.ArrayList.add(ArrayList.java:367)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.AbstractList.add(AbstractList.java:91)
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:74)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1139)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:260)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:234)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status

[Jonathan Hsieh (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh resolved HBASE-5675.
---

Resolution: Duplicate

> Create table fails if we keep refreshing master's UI for task monitor status
> 
>
> Key: HBASE-5675
> URL: https://issues.apache.org/jira/browse/HBASE-5675
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Mubarak Seyed
>Assignee: Mubarak Seyed
>  Labels: noob
>
> I tried to create a table with 2K pre-split regions, region assignment was in 
> middle and i was keep refreshing master's web UI to find the status of the 
> task using task monitor, table creation was failed and {{META}} was showing 
> 2K regions with server location value is {{null}} and regions weren't 
> deployed onto region-servers.
> {code}
> table_ACreating table table_A
> java.io.IOException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>   at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>   at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
>   at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
>   at com.test.tools.hbase.schema.main(schema.java:627)
> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at java.util.SubList.checkForComodification(AbstractList.java:752)
>   at java.util.SubList.add(AbstractList.java:632)
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.SubList.add(AbstractList.java:633)
>   ..
>   ..
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.AbstractList.add(AbstractList.java:91)
>   at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
>   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>   at $Proxy5.createTable(Unknown Source)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
>   
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5667) RegexStringComparator supports java.util.regex.Pattern flags

[Jimmy Xiang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241870#comment-13241870
 ] 

Jimmy Xiang commented on HBASE-5667:


For this patch, it changes the constructor of RegexStringComparator.  A Pattern 
is hard to be pb'd.  Can we specify the flags in a different way,
for example, using string, and/or some primitive parameters?

> RegexStringComparator supports java.util.regex.Pattern flags
> 
>
> Key: HBASE-5667
> URL: https://issues.apache.org/jira/browse/HBASE-5667
> Project: HBase
>  Issue Type: Improvement
>  Components: filters
>Reporter: David Arthur
>Priority: Minor
> Attachments: HBASE-5667.diff
>
>
> * Add constructor that takes in a Pattern
> * Add Pattern's flags to Writable fields, and actually use them when 
> recomposing the Filter

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status

[Jonathan Hsieh (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-5675:
--

Affects Version/s: (was: 0.94.0)

> Create table fails if we keep refreshing master's UI for task monitor status
> 
>
> Key: HBASE-5675
> URL: https://issues.apache.org/jira/browse/HBASE-5675
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Mubarak Seyed
>Assignee: Mubarak Seyed
>  Labels: noob
>
> I tried to create a table with 2K pre-split regions, region assignment was in 
> middle and i was keep refreshing master's web UI to find the status of the 
> task using task monitor, table creation was failed and {{META}} was showing 
> 2K regions with server location value is {{null}} and regions weren't 
> deployed onto region-servers.
> {code}
> table_ACreating table table_A
> java.io.IOException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>   at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>   at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
>   at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
>   at com.test.tools.hbase.schema.main(schema.java:627)
> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at java.util.SubList.checkForComodification(AbstractList.java:752)
>   at java.util.SubList.add(AbstractList.java:632)
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.SubList.add(AbstractList.java:633)
>   ..
>   ..
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.AbstractList.add(AbstractList.java:91)
>   at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
>   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>   at $Proxy5.createTable(Unknown Source)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
>   
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241866#comment-13241866
 ] 

Jonathan Hsieh commented on HBASE-5675:
---

The fix was in HBASE-5535 and in the 0.94 branch and above.  I'll mark this 
issue as duplicate, and get it added into the issues to fix in CDH. 

> Create table fails if we keep refreshing master's UI for task monitor status
> 
>
> Key: HBASE-5675
> URL: https://issues.apache.org/jira/browse/HBASE-5675
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.4, 0.92.0, 0.94.0
>Reporter: Mubarak Seyed
>Assignee: Mubarak Seyed
>  Labels: noob
>
> I tried to create a table with 2K pre-split regions, region assignment was in 
> middle and i was keep refreshing master's web UI to find the status of the 
> task using task monitor, table creation was failed and {{META}} was showing 
> 2K regions with server location value is {{null}} and regions weren't 
> deployed onto region-servers.
> {code}
> table_ACreating table table_A
> java.io.IOException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>   at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>   at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294)
>   at com.test.tools.hbase.schema.createIfNotExists(schema.java:520)
>   at com.test.tools.hbase.schema.main(schema.java:627)
> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
> java.util.ConcurrentModificationException
>   at java.util.SubList.checkForComodification(AbstractList.java:752)
>   at java.util.SubList.add(AbstractList.java:632)
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.SubList.add(AbstractList.java:633)
>   ..
>   ..
>   at java.util.SubList.add(AbstractList.java:633)
>   at java.util.AbstractList.add(AbstractList.java:91)
>   at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510)
>   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813)
>   at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780)
>   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>   at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>   at $Proxy5.createTable(Unknown Source)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382)
>   
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Enis Soztutar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241863#comment-13241863
 ] 

Enis Soztutar commented on HBASE-5619:
--

I think for compiling protoc installation should not be a dependency. Why don't 
we add the generated java files to the source tree, so that only if you change 
the protoc definitions, you would have to recompile the .proto files. As far as 
I know, this is how it is done for thrift dependencies in other projects. 

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5667) RegexStringComparator supports java.util.regex.Pattern flags

[Jimmy Xiang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241861#comment-13241861
 ] 

Jimmy Xiang commented on HBASE-5667:


@Stack, I prefer to change them to pb so we should not bother to make them 
VersionedWritables for now.
We have lots of filters.  We need to abstract them out and have a generic way 
to define them in pb.


> RegexStringComparator supports java.util.regex.Pattern flags
> 
>
> Key: HBASE-5667
> URL: https://issues.apache.org/jira/browse/HBASE-5667
> Project: HBase
>  Issue Type: Improvement
>  Components: filters
>Reporter: David Arthur
>Priority: Minor
> Attachments: HBASE-5667.diff
>
>
> * Add constructor that takes in a Pattern
> * Add Pattern's flags to Writable fields, and actually use them when 
> recomposing the Filter

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5667) RegexStringComparator supports java.util.regex.Pattern flags

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241835#comment-13241835
 ] 

stack commented on HBASE-5667:
--

@Jimmy Whats going to happen in 0.96 serializing filters?  Will these be pb'd?  
Or should we go to the bother of making all filters VersionedWritables?

> RegexStringComparator supports java.util.regex.Pattern flags
> 
>
> Key: HBASE-5667
> URL: https://issues.apache.org/jira/browse/HBASE-5667
> Project: HBase
>  Issue Type: Improvement
>  Components: filters
>Reporter: David Arthur
>Priority: Minor
> Attachments: HBASE-5667.diff
>
>
> * Add constructor that takes in a Pattern
> * Add Pattern's flags to Writable fields, and actually use them when 
> recomposing the Filter

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Jimmy Xiang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241819#comment-13241819
 ] 

Jimmy Xiang commented on HBASE-5619:


@Stack, so far, I could not find a good protoc maven plugin. I don't remember I 
tried to install it on my Ubuntu.
That's the download site for protobuf compiler: 
http://code.google.com/p/protobuf/downloads/list
But for Linux, I think it is easy to install with rpm/apt-get.

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5655) Cap space usage of default log4j rolling policy

[Himanshu Vashishtha (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Himanshu Vashishtha updated HBASE-5655:
---

Attachment: HBase-5655-v2.patch

Add a brief note about how to roll back to the old log rolling policy in the 
hbase-env.sh file. Also mention this jira number for more reference.

> Cap space usage of default log4j rolling policy
> ---
>
> Key: HBASE-5655
> URL: https://issues.apache.org/jira/browse/HBASE-5655
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.1
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Attachments: 5655-v1.patch, HBase-5655-v2.patch
>
>
> The current default log4j policy is to use Daily Rolling File Appender 
> (DRFA). At times, its good to have a cap on the maximum size of the logs in 
> order to limit its disk usage. Here is a proposal to set a new file appemder 
> (RFA) as the default appender. It can be configured via env so that existing 
> tools can use the current behavior of using DRFA instead. 
> This is in parallel with jira Hadoop-8149.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241801#comment-13241801
 ] 

jirapos...@reviews.apache.org commented on HBASE-5451:
--



bq.  On 2012-03-29 18:13:02, Benoit Sigoure wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 102
bq.  > 
bq.  >
bq.  > Devaraj, the problem with up'ing the version number is that it makes 
the life of backwards-compatible clients like asynchbase even more difficult 
than it already is.
bq.  > 
bq.  > I proposed another idea to Stack, I don't know if he shared it with 
you, so here it is:  In the znode used to store the -ROOT- region, add the 
protocol version number (make it 5 there if you want).  This way clients that 
are finding where -ROOT- is will be able to figure out the protocol version to 
use *before* connecting to -ROOT-.
bq.  > 
bq.  > This better than parsing the string of the VersionMismatchException, 
which you said yourself is hacky (and also inefficient), so we don't wanna do 
that.
bq.  
bq.  Devaraj Das wrote:
bq.  Thanks, Benoit for getting back. The idea you have is cool but I don't 
see how that'd apply to old existing clients. In the asynchbase case, one would 
have to write new code to take care of the proposed arrangement, right? Am I 
missing something?
bq.  
bq.  I'll upload a patch shortly that doesn't change the version number in 
the RPC... (we can revisit this issue later).
bq.  
bq.  Michael Stack wrote:
bq.  Deveraj: I talked w/ B.  It makes sense that you be able to find the 
'version' of an hbase cluster, or at least, the version that a client should 
use when it goes to read the root/meta region content by looking in zk.  I 
intend to remove root for 0.96.0.  I also intend to change how all is 
serialized to zk in 0.96 to make it pb based.   When I change the 
root-region-location in zk, I'll include version a client needs reading 
(talking w/ Benoit, rather than remove this znode, we should probably just keep 
it only have it point at .META. from here on out; i.e. meta becomes the root.. 
but that is for another issue).
bq.  
bq.  So, go ahead, please change the version in your patch.  Sorry for the 
distraction.

Thanks, Stack, for helping on getting a resolution!

Is there a jira on the topic of removing root for 0.96.0 ?


- Devaraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4096/#review6525
---


On 2012-03-01 03:40:14, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4096/
bq.  ---
bq.  
bq.  (Updated 2012-03-01 03:40:14)
bq.  
bq.  
bq.  Review request for .
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Switch RPC call envelope/headers to PBs
bq.  
bq.  
bq.  This addresses bug HBASE-5451.
bq.  https://issues.apache.org/jira/browse/HBASE-5451
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/proto/RPCMessageProto.proto
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4096/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Devaraj
bq.  
bq.



> Switch RPC call envelope/headers to PBs
> ---
>
> Key: HBASE-5451
> URL: https://issues.apache.org/jira/browse/HBASE-5451
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Affects Versions: 0.94.0
>Reporter: Todd Lipcon
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more informatio

[jira] [Commented] (HBASE-4398) If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml.

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241792#comment-13241792
 ] 

stack commented on HBASE-4398:
--

Thanks for the patch Takuya.  I applied it to 0.92, 0.94, and trunk.

> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> -
>
> Key: HBASE-4398
> URL: https://issues.apache.org/jira/browse/HBASE-4398
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.90.4
>Reporter: Takuya Ueshin
>Assignee: Takuya Ueshin
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-4398.patch
>
>
> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> We can reproduce the problem by the following instructions:
> {noformat}
> - Add HRegionPartitioner.class to the 4th argument of
> TableMapReduceUtil#initTableReducerJob()
> at line around 133
> in src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
> - Change or remove "hbase.zookeeper.property.clientPort" property
> in hbase-site.xml ( for example, changed to 12345 ).
> - run testMultiRegionTable()
> {noformat}
> Then I got error messages as following:
> {noformat}
> 2011-09-12 22:28:51,020 DEBUG [Thread-832] zookeeper.ZKUtil(93): hconnection 
> opening connection to ZooKeeper with ensemble (localhost:12345)
> 2011-09-12 22:28:51,022 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(89): The identifier of this process is 
> 43200@imac.local
> 2011-09-12 22:28:51,123 WARN  [Thread-832] 
> zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: 
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for /hbase/master
> 2011-09-12 22:28:51,123 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after 
> sleeping 1000 ms
>  =
> 2011-09-12 22:29:02,418 ERROR [Thread-832] mapreduce.HRegionPartitioner(125): 
> java.io.IOException: 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2e54e48d
>  closed
> 2011-09-12 22:29:02,422 WARN  [Thread-832] mapred.LocalJobRunner$Job(256): 
> job_local_0001
> java.lang.NullPointerException
>at 
> org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:128)
>at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:527)
>at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> {noformat}
> I think HTable should connect to ZooKeeper at port 21818 configured at client 
> side instead of 12345 in hbase-site.xml
> and It might be caused by "HBaseConfiguration.addHbaseResources(conf);" in 
> HRegionPartitioner#setConf(Configuration).
> And this might mean that all of client side configurations, also configured 
> in hbase-site.xml, are overwritten caused by this problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4398) If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml.

[stack (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4398.
--

   Resolution: Fixed
Fix Version/s: 0.94.1
   0.92.2
 Assignee: Takuya Ueshin
 Hadoop Flags: Reviewed

> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> -
>
> Key: HBASE-4398
> URL: https://issues.apache.org/jira/browse/HBASE-4398
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.90.4
>Reporter: Takuya Ueshin
>Assignee: Takuya Ueshin
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-4398.patch
>
>
> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> We can reproduce the problem by the following instructions:
> {noformat}
> - Add HRegionPartitioner.class to the 4th argument of
> TableMapReduceUtil#initTableReducerJob()
> at line around 133
> in src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
> - Change or remove "hbase.zookeeper.property.clientPort" property
> in hbase-site.xml ( for example, changed to 12345 ).
> - run testMultiRegionTable()
> {noformat}
> Then I got error messages as following:
> {noformat}
> 2011-09-12 22:28:51,020 DEBUG [Thread-832] zookeeper.ZKUtil(93): hconnection 
> opening connection to ZooKeeper with ensemble (localhost:12345)
> 2011-09-12 22:28:51,022 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(89): The identifier of this process is 
> 43200@imac.local
> 2011-09-12 22:28:51,123 WARN  [Thread-832] 
> zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: 
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for /hbase/master
> 2011-09-12 22:28:51,123 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after 
> sleeping 1000 ms
>  =
> 2011-09-12 22:29:02,418 ERROR [Thread-832] mapreduce.HRegionPartitioner(125): 
> java.io.IOException: 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2e54e48d
>  closed
> 2011-09-12 22:29:02,422 WARN  [Thread-832] mapred.LocalJobRunner$Job(256): 
> job_local_0001
> java.lang.NullPointerException
>at 
> org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:128)
>at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:527)
>at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> {noformat}
> I think HTable should connect to ZooKeeper at port 21818 configured at client 
> side instead of 12345 in hbase-site.xml
> and It might be caused by "HBaseConfiguration.addHbaseResources(conf);" in 
> HRegionPartitioner#setConf(Configuration).
> And this might mean that all of client side configurations, also configured 
> in hbase-site.xml, are overwritten caused by this problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4398) If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml.

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241787#comment-13241787
 ] 

stack commented on HBASE-4398:
--

Good one Takuya.  Thats broke.  I would suggest that your test case while 
comprehensive, it is way overkill for your fix.  Its also mostly a copy of 
another heavy-weight test we already run.   I can't believe I'm saying this but 
I'm going to commit this fix w/o including the test.

> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> -
>
> Key: HBASE-4398
> URL: https://issues.apache.org/jira/browse/HBASE-4398
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.90.4
>Reporter: Takuya Ueshin
> Attachments: HBASE-4398.patch
>
>
> If HRegionPartitioner is used in MapReduce, client side configurations are 
> overwritten by hbase-site.xml.
> We can reproduce the problem by the following instructions:
> {noformat}
> - Add HRegionPartitioner.class to the 4th argument of
> TableMapReduceUtil#initTableReducerJob()
> at line around 133
> in src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
> - Change or remove "hbase.zookeeper.property.clientPort" property
> in hbase-site.xml ( for example, changed to 12345 ).
> - run testMultiRegionTable()
> {noformat}
> Then I got error messages as following:
> {noformat}
> 2011-09-12 22:28:51,020 DEBUG [Thread-832] zookeeper.ZKUtil(93): hconnection 
> opening connection to ZooKeeper with ensemble (localhost:12345)
> 2011-09-12 22:28:51,022 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(89): The identifier of this process is 
> 43200@imac.local
> 2011-09-12 22:28:51,123 WARN  [Thread-832] 
> zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: 
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for /hbase/master
> 2011-09-12 22:28:51,123 INFO  [Thread-832] 
> zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after 
> sleeping 1000 ms
>  =
> 2011-09-12 22:29:02,418 ERROR [Thread-832] mapreduce.HRegionPartitioner(125): 
> java.io.IOException: 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2e54e48d
>  closed
> 2011-09-12 22:29:02,422 WARN  [Thread-832] mapred.LocalJobRunner$Job(256): 
> job_local_0001
> java.lang.NullPointerException
>at 
> org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:128)
>at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:527)
>at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> {noformat}
> I think HTable should connect to ZooKeeper at port 21818 configured at client 
> side instead of 12345 in hbase-site.xml
> and It might be caused by "HBaseConfiguration.addHbaseResources(conf);" in 
> HRegionPartitioner#setConf(Configuration).
> And this might mean that all of client side configurations, also configured 
> in hbase-site.xml, are overwritten caused by this problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241788#comment-13241788
 ] 

Hudson commented on HBASE-5097:
---

Integrated in HBase-0.92 #342 (See 
[https://builds.apache.org/job/HBase-0.92/342/])
HBASE-5097 RegionObserver implementation whose preScannerOpen and 
postScannerOpen Impl return null can stall the system initialization through 
NPE (Ram) (Revision 1307034)

 Result = SUCCESS
ramkrishna : 
Files : 
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


> RegionObserver implementation whose preScannerOpen and postScannerOpen Impl 
> return null can stall the system initialization through NPE
> ---
>
> Key: HBASE-5097
> URL: https://issues.apache.org/jira/browse/HBASE-5097
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5097.patch, HBASE-5097_1.patch, HBASE-5097_2.patch
>
>
> In HRegionServer.java openScanner()
> {code}
>   r.prepareScanner(scan);
>   RegionScanner s = null;
>   if (r.getCoprocessorHost() != null) {
> s = r.getCoprocessorHost().preScannerOpen(scan);
>   }
>   if (s == null) {
> s = r.getScanner(scan);
>   }
>   if (r.getCoprocessorHost() != null) {
> s = r.getCoprocessorHost().postScannerOpen(scan, s);
>   }
> {code}
> If we dont have implemention for postScannerOpen the RegionScanner is null 
> and so throwing nullpointer 
> {code}
> java.lang.NullPointerException
>   at 
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
> {code}
> Making this defect as blocker.. Pls feel free to change the priority if am 
> wrong.  Also correct me if my way of trying out coprocessors without 
> implementing postScannerOpen is wrong.  Am just a learner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241778#comment-13241778
 ] 

jirapos...@reviews.apache.org commented on HBASE-5451:
--



bq.  On 2012-03-29 18:13:02, Benoit Sigoure wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 102
bq.  > 
bq.  >
bq.  > Devaraj, the problem with up'ing the version number is that it makes 
the life of backwards-compatible clients like asynchbase even more difficult 
than it already is.
bq.  > 
bq.  > I proposed another idea to Stack, I don't know if he shared it with 
you, so here it is:  In the znode used to store the -ROOT- region, add the 
protocol version number (make it 5 there if you want).  This way clients that 
are finding where -ROOT- is will be able to figure out the protocol version to 
use *before* connecting to -ROOT-.
bq.  > 
bq.  > This better than parsing the string of the VersionMismatchException, 
which you said yourself is hacky (and also inefficient), so we don't wanna do 
that.
bq.  
bq.  Devaraj Das wrote:
bq.  Thanks, Benoit for getting back. The idea you have is cool but I don't 
see how that'd apply to old existing clients. In the asynchbase case, one would 
have to write new code to take care of the proposed arrangement, right? Am I 
missing something?
bq.  
bq.  I'll upload a patch shortly that doesn't change the version number in 
the RPC... (we can revisit this issue later).

Deveraj: I talked w/ B.  It makes sense that you be able to find the 'version' 
of an hbase cluster, or at least, the version that a client should use when it 
goes to read the root/meta region content by looking in zk.  I intend to remove 
root for 0.96.0.  I also intend to change how all is serialized to zk in 0.96 
to make it pb based.   When I change the root-region-location in zk, I'll 
include version a client needs reading (talking w/ Benoit, rather than remove 
this znode, we should probably just keep it only have it point at .META. from 
here on out; i.e. meta becomes the root.. but that is for another issue).

So, go ahead, please change the version in your patch.  Sorry for the 
distraction.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4096/#review6525
---


On 2012-03-01 03:40:14, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4096/
bq.  ---
bq.  
bq.  (Updated 2012-03-01 03:40:14)
bq.  
bq.  
bq.  Review request for .
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Switch RPC call envelope/headers to PBs
bq.  
bq.  
bq.  This addresses bug HBASE-5451.
bq.  https://issues.apache.org/jira/browse/HBASE-5451
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/proto/RPCMessageProto.proto
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4096/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Devaraj
bq.  
bq.



> Switch RPC call envelope/headers to PBs
> ---
>
> Key: HBASE-5451
> URL: https://issues.apache.org/jira/browse/HBASE-5451
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Affects Versions: 0.94.0
>Reporter: Todd Lipcon
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4393) Implement a canary monitoring program

[Jonathan Hsieh (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh reassigned HBASE-4393:
-

Assignee: Matteo Bertozzi  (was: Amandeep Khurana)

> Implement a canary monitoring program
> -
>
> Key: HBASE-4393
> URL: https://issues.apache.org/jira/browse/HBASE-4393
> Project: HBase
>  Issue Type: New Feature
>  Components: monitoring
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Matteo Bertozzi
> Attachments: Canary-v0.java, HBaseCanary.java
>
>
> This JIRA is to implement a standalone program that can be used to do "canary 
> monitoring" of a running HBase cluster. This program would gather a list of 
> the regions in the cluster, then iterate over them doing lightweight 
> operations (eg short scans) to provide metrics about latency as well as alert 
> on availability issues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241769#comment-13241769
 ] 

jirapos...@reviews.apache.org commented on HBASE-5451:
--



bq.  On 2012-03-29 18:13:02, Benoit Sigoure wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java,
 line 102
bq.  > 
bq.  >
bq.  > Devaraj, the problem with up'ing the version number is that it makes 
the life of backwards-compatible clients like asynchbase even more difficult 
than it already is.
bq.  > 
bq.  > I proposed another idea to Stack, I don't know if he shared it with 
you, so here it is:  In the znode used to store the -ROOT- region, add the 
protocol version number (make it 5 there if you want).  This way clients that 
are finding where -ROOT- is will be able to figure out the protocol version to 
use *before* connecting to -ROOT-.
bq.  > 
bq.  > This better than parsing the string of the VersionMismatchException, 
which you said yourself is hacky (and also inefficient), so we don't wanna do 
that.

Thanks, Benoit for getting back. The idea you have is cool but I don't see how 
that'd apply to old existing clients. In the asynchbase case, one would have to 
write new code to take care of the proposed arrangement, right? Am I missing 
something?

I'll upload a patch shortly that doesn't change the version number in the 
RPC... (we can revisit this issue later).


- Devaraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4096/#review6525
---


On 2012-03-01 03:40:14, Devaraj Das wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4096/
bq.  ---
bq.  
bq.  (Updated 2012-03-01 03:40:14)
bq.  
bq.  
bq.  Review request for .
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Switch RPC call envelope/headers to PBs
bq.  
bq.  
bq.  This addresses bug HBASE-5451.
bq.  https://issues.apache.org/jira/browse/HBASE-5451
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java
 1294899 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/proto/RPCMessageProto.proto
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4096/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Devaraj
bq.  
bq.



> Switch RPC call envelope/headers to PBs
> ---
>
> Key: HBASE-5451
> URL: https://issues.apache.org/jira/browse/HBASE-5451
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Affects Versions: 0.94.0
>Reporter: Todd Lipcon
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241765#comment-13241765
 ] 

stack commented on HBASE-5665:
--

Nice test Cosmin

> Repeated split causes HRegionServer failures and breaks table 
> --
>
> Key: HBASE-5665
> URL: https://issues.apache.org/jira/browse/HBASE-5665
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0, 0.92.1
>Reporter: Cosmin Lehene
>Assignee: Cosmin Lehene
>Priority: Blocker
> Attachments: HBASE-5665-0.92.patch
>
>
> Repeated splits on large tables (2 consecutive would suffice) will 
> essentially "break" the table (and the cluster), unrecoverable.
> The regionserver doing the split dies and the master will get into an 
> infinite loop trying to assign regions that seem to have the files missing 
> from HDFS.
> The table can be disabled once. upon trying to re-enable it, it will remain 
> in an intermediary state forever.
> I was able to reproduce this on a smaller table consistently.
> {code}
> hbase(main):030:0> (0..1).each{|x| put 't1', "#{x}", 'f1:t', 'dd'}
> hbase(main):030:0> (0..1000).each{|x| split 't1', "#{x*10}"}
> {code}
> Running overlapping splits in parallel (e.g. "#{x*10+1}", "#{x*10+2}"... ) 
> will reproduce the issue almost instantly and consistently. 
> {code}
> 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: 
> Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in 
> META
> 2012-03-28 10:57:16,321 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for 
> t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1..  
> compaction_queue=(0:1), split_queue=10
> 2012-03-28 10:57:16,343 INFO 
> org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup 
> of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; 
> Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> java.io.IOException: Failed 
> ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451)
> at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException: File does not exist: 
> /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1008)
> at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:65)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548)
> at 
> org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284)
> at org.apache.hadoop.hbase.regionserver.Store.(Store.java:221)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484)
> ... 1 more
> 2012-03-28 10:57:16,345 FATAL 
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server 
> ld2,60020,1332957343833: Abort; we got an error after point-of-no-return
> {code}
> http://hastebin.com/diqinibajo.avrasm
> later edit:
> (I'm using the last 4 characters from each string)
> Region 94e3 has storefile 7237
> Region 94e3 gets splited in daughter

[jira] [Commented] (HBASE-4348) Add metrics for regions in transition

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241761#comment-13241761
 ] 

Jonathan Hsieh commented on HBASE-4348:
---

Himanshu, for us to commit patches, we need you to attach the patch to the JIRA 
and the grant permission.  Currently,  review board is optional, but makes it 
way easier for us to review.  

Please do so after you address the latest comments.  Thanks!

> Add metrics for regions in transition
> -
>
> Key: HBASE-4348
> URL: https://issues.apache.org/jira/browse/HBASE-4348
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Himanshu Vashishtha
>Priority: Minor
>  Labels: noob
> Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, 
> RITs.png, RegionInTransitions2.png, metrics-v2.patch
>
>
> The following metrics would be useful for monitoring the master:
> - the number of regions in transition
> - the number of regions in transition that have been in transition for more 
> than a minute
> - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241756#comment-13241756
 ] 

stack commented on HBASE-5619:
--

@Jimmy Is there a maven plugin for protoc so folks don't have to install it?  
At least, we should be checking for its presence and complaining w/ a pointer 
to a how-to-install doc if not present?  Is that possible?

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4348) Add metrics for regions in transition

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241753#comment-13241753
 ] 

jirapos...@reviews.apache.org commented on HBASE-4348:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4402/#review6540
---



src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


Ok, but what is the rationale behind having such a metrics?



src/main/java/org/apache/hadoop/hbase/master/HMaster.java


Ok. I will rename it.


- Himanshu


On 2012-03-29 15:57:19, Himanshu Vashishtha wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4402/
bq.  ---
bq.  
bq.  (Updated 2012-03-29 15:57:19)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch is for adding Region in transition metrics to the HMaster 
metrics system. It also adds these metrics in the master ui, in the Region in 
transition section. I have attached the proposed new format in the jira 4348.
bq.  
bq.  
bq.  This addresses bug HBase-4348.
bq.  https://issues.apache.org/jira/browse/HBase-4348
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon
 0dc0691 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 19be4de 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
ae468ca 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java c4b4d30 
bq.src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 
83abc52 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
d68ce33 
bq.  
bq.  Diff: https://reviews.apache.org/r/4402/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran on a 5 node cluster and kill region servers randomly to observe the 
changes in the RIT metrics as emitted out by the Master's mxbean;
bq.  
bq.  mvn test passes without any failure.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Himanshu
bq.  
bq.



> Add metrics for regions in transition
> -
>
> Key: HBASE-4348
> URL: https://issues.apache.org/jira/browse/HBASE-4348
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Himanshu Vashishtha
>Priority: Minor
>  Labels: noob
> Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, 
> RITs.png, RegionInTransitions2.png, metrics-v2.patch
>
>
> The following metrics would be useful for monitoring the master:
> - the number of regions in transition
> - the number of regions in transition that have been in transition for more 
> than a minute
> - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Jimmy Xiang (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241622#comment-13241622
 ] 

Jimmy Xiang commented on HBASE-5619:


@Stack, could you please install protoc and give it a try again?
>From now on, we need protoc to compile. :)

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241611#comment-13241611
 ] 

Hadoop QA commented on HBASE-5665:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12520458/HBASE-5665-0.92.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1347//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1347//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1347//console

This message is automatically generated.

> Repeated split causes HRegionServer failures and breaks table 
> --
>
> Key: HBASE-5665
> URL: https://issues.apache.org/jira/browse/HBASE-5665
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0, 0.92.1
>Reporter: Cosmin Lehene
>Assignee: Cosmin Lehene
>Priority: Blocker
> Attachments: HBASE-5665-0.92.patch
>
>
> Repeated splits on large tables (2 consecutive would suffice) will 
> essentially "break" the table (and the cluster), unrecoverable.
> The regionserver doing the split dies and the master will get into an 
> infinite loop trying to assign regions that seem to have the files missing 
> from HDFS.
> The table can be disabled once. upon trying to re-enable it, it will remain 
> in an intermediary state forever.
> I was able to reproduce this on a smaller table consistently.
> {code}
> hbase(main):030:0> (0..1).each{|x| put 't1', "#{x}", 'f1:t', 'dd'}
> hbase(main):030:0> (0..1000).each{|x| split 't1', "#{x*10}"}
> {code}
> Running overlapping splits in parallel (e.g. "#{x*10+1}", "#{x*10+2}"... ) 
> will reproduce the issue almost instantly and consistently. 
> {code}
> 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: 
> Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in 
> META
> 2012-03-28 10:57:16,321 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for 
> t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1..  
> compaction_queue=(0:1), split_queue=10
> 2012-03-28 10:57:16,343 INFO 
> org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup 
> of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; 
> Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> java.io.IOException: Failed 
> ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451)
> at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException: File does not exist: 
> /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1008)
> at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:65)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(S

[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241607#comment-13241607
 ] 

Hudson commented on HBASE-5638:
---

Integrated in HBase-0.94 #66 (See 
[https://builds.apache.org/job/HBase-0.94/66/])
HBASE-5638 Readability improvements on HBASE-5633: NPE reading ZK config in 
HBase (Matteo Bertozzi) (Revision 1307086)

 Result = SUCCESS
jmhsieh : 
Files : 
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java


> Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
> --
>
> Key: HBASE-5638
> URL: https://issues.apache.org/jira/browse/HBASE-5638
> Project: HBase
>  Issue Type: Sub-task
>  Components: zookeeper
>Affects Versions: 0.90.6, 0.92.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.94.1
>
> Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
> HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, 
> HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5633) NPE reading ZK config in HBase

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241606#comment-13241606
 ] 

Hudson commented on HBASE-5633:
---

Integrated in HBase-0.94 #66 (See 
[https://builds.apache.org/job/HBase-0.94/66/])
HBASE-5638 Readability improvements on HBASE-5633: NPE reading ZK config in 
HBase (Matteo Bertozzi) (Revision 1307086)

 Result = SUCCESS
jmhsieh : 
Files : 
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java


> NPE reading ZK config in HBase
> --
>
> Key: HBASE-5633
> URL: https://issues.apache.org/jira/browse/HBASE-5633
> Project: HBase
>  Issue Type: Bug
>  Components: zookeeper
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
> HBASE-5633-v1.patch, HBASE-5633-v2.patch
>
>
> If zoo.cfg contains server.* ("server.0=server0:2888:3888\n") and 
> cluster.distributed property (in hbase-site.xml) is empty we get an NPE in 
> parseZooCfg().
> The easy way to reproduce the bug is running 
> org.apache.hbase.zookeeper.TestHQuorumPeer with hbase-site.xml containing:
> {code}
> 
>   hbase.cluster.distributed
>   
> 
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4348) Add metrics for regions in transition

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241604#comment-13241604
 ] 

jirapos...@reviews.apache.org commented on HBASE-4348:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4402/#review6538
---



src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


Can we have a metric for how long this routine takes ?




src/main/java/org/apache/hadoop/hbase/master/HMaster.java


lastMsg makes me think of the last message.
How about naming this variable lastMsgTs or something similar ?


- Ted


On 2012-03-29 15:57:19, Himanshu Vashishtha wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4402/
bq.  ---
bq.  
bq.  (Updated 2012-03-29 15:57:19)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch is for adding Region in transition metrics to the HMaster 
metrics system. It also adds these metrics in the master ui, in the Region in 
transition section. I have attached the proposed new format in the jira 4348.
bq.  
bq.  
bq.  This addresses bug HBase-4348.
bq.  https://issues.apache.org/jira/browse/HBase-4348
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon
 0dc0691 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 19be4de 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
ae468ca 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java c4b4d30 
bq.src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 
83abc52 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
d68ce33 
bq.  
bq.  Diff: https://reviews.apache.org/r/4402/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran on a 5 node cluster and kill region servers randomly to observe the 
changes in the RIT metrics as emitted out by the Master's mxbean;
bq.  
bq.  mvn test passes without any failure.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Himanshu
bq.  
bq.



> Add metrics for regions in transition
> -
>
> Key: HBASE-4348
> URL: https://issues.apache.org/jira/browse/HBASE-4348
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Himanshu Vashishtha
>Priority: Minor
>  Labels: noob
> Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, 
> RITs.png, RegionInTransitions2.png, metrics-v2.patch
>
>
> The following metrics would be useful for monitoring the master:
> - the number of regions in transition
> - the number of regions in transition that have been in transition for more 
> than a minute
> - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241602#comment-13241602
 ] 

Hadoop QA commented on HBASE-5638:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12520493/HBASE-5638-trunk-v2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestImportTsv
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1346//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1346//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1346//console

This message is automatically generated.

> Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
> --
>
> Key: HBASE-5638
> URL: https://issues.apache.org/jira/browse/HBASE-5638
> Project: HBase
>  Issue Type: Sub-task
>  Components: zookeeper
>Affects Versions: 0.90.6, 0.92.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.94.1
>
> Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
> HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, 
> HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4348) Add metrics for regions in transition

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241600#comment-13241600
 ] 

jirapos...@reviews.apache.org commented on HBASE-4348:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4402/#review6537
---

Ship it!


lgtm.  I'll fix the spacing nits as I commit.


src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


nit: spacing



src/main/java/org/apache/hadoop/hbase/master/HMaster.java


spacing



src/main/java/org/apache/hadoop/hbase/master/HMaster.java


spacing


- jmhsieh


On 2012-03-29 15:57:19, Himanshu Vashishtha wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4402/
bq.  ---
bq.  
bq.  (Updated 2012-03-29 15:57:19)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch is for adding Region in transition metrics to the HMaster 
metrics system. It also adds these metrics in the master ui, in the Region in 
transition section. I have attached the proposed new format in the jira 4348.
bq.  
bq.  
bq.  This addresses bug HBase-4348.
bq.  https://issues.apache.org/jira/browse/HBase-4348
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon
 0dc0691 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 19be4de 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
ae468ca 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java c4b4d30 
bq.src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 
83abc52 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
d68ce33 
bq.  
bq.  Diff: https://reviews.apache.org/r/4402/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran on a 5 node cluster and kill region servers randomly to observe the 
changes in the RIT metrics as emitted out by the Master's mxbean;
bq.  
bq.  mvn test passes without any failure.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Himanshu
bq.  
bq.



> Add metrics for regions in transition
> -
>
> Key: HBASE-4348
> URL: https://issues.apache.org/jira/browse/HBASE-4348
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Himanshu Vashishtha
>Priority: Minor
>  Labels: noob
> Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, 
> RITs.png, RegionInTransitions2.png, metrics-v2.patch
>
>
> The following metrics would be useful for monitoring the master:
> - the number of regions in transition
> - the number of regions in transition that have been in transition for more 
> than a minute
> - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[Zhihong Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241594#comment-13241594
 ] 

Zhihong Yu commented on HBASE-5619:
---

@Stack:
Was protoc installed on the box you performed the build.

I did a clean build after applying patch v4 and didn't see the above error.

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table

[Cosmin Lehene (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241593#comment-13241593
 ] 

Cosmin Lehene commented on HBASE-5665:
--

BTW - I don't think getSplitPoint should do that check, and we also shouldn't 
have to places where we check for references - perhaps we should have another 
JIRA to fix this in trunk?

> Repeated split causes HRegionServer failures and breaks table 
> --
>
> Key: HBASE-5665
> URL: https://issues.apache.org/jira/browse/HBASE-5665
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0, 0.92.1
>Reporter: Cosmin Lehene
>Assignee: Cosmin Lehene
>Priority: Blocker
> Attachments: HBASE-5665-0.92.patch
>
>
> Repeated splits on large tables (2 consecutive would suffice) will 
> essentially "break" the table (and the cluster), unrecoverable.
> The regionserver doing the split dies and the master will get into an 
> infinite loop trying to assign regions that seem to have the files missing 
> from HDFS.
> The table can be disabled once. upon trying to re-enable it, it will remain 
> in an intermediary state forever.
> I was able to reproduce this on a smaller table consistently.
> {code}
> hbase(main):030:0> (0..1).each{|x| put 't1', "#{x}", 'f1:t', 'dd'}
> hbase(main):030:0> (0..1000).each{|x| split 't1', "#{x*10}"}
> {code}
> Running overlapping splits in parallel (e.g. "#{x*10+1}", "#{x*10+2}"... ) 
> will reproduce the issue almost instantly and consistently. 
> {code}
> 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: 
> Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in 
> META
> 2012-03-28 10:57:16,321 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for 
> t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1..  
> compaction_queue=(0:1), split_queue=10
> 2012-03-28 10:57:16,343 INFO 
> org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup 
> of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; 
> Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> java.io.IOException: Failed 
> ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451)
> at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException: File does not exist: 
> /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1008)
> at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:65)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548)
> at 
> org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284)
> at org.apache.hadoop.hbase.regionserver.Store.(Store.java:221)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484)
> ... 1 more
> 2012-03-28 10:57:16,345 FATAL 
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server 
> ld2,60020,1332957343833: Abort; we got an error after point-of-no-return
> {

[jira] [Updated] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table

[Cosmin Lehene (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cosmin Lehene updated HBASE-5665:
-

Affects Version/s: (was: 0.94.1)
   (was: 0.96.0)
   (was: 0.94.0)
   Status: Patch Available  (was: Open)

> Repeated split causes HRegionServer failures and breaks table 
> --
>
> Key: HBASE-5665
> URL: https://issues.apache.org/jira/browse/HBASE-5665
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.1, 0.92.0
>Reporter: Cosmin Lehene
>Assignee: Cosmin Lehene
>Priority: Blocker
> Attachments: HBASE-5665-0.92.patch
>
>
> Repeated splits on large tables (2 consecutive would suffice) will 
> essentially "break" the table (and the cluster), unrecoverable.
> The regionserver doing the split dies and the master will get into an 
> infinite loop trying to assign regions that seem to have the files missing 
> from HDFS.
> The table can be disabled once. upon trying to re-enable it, it will remain 
> in an intermediary state forever.
> I was able to reproduce this on a smaller table consistently.
> {code}
> hbase(main):030:0> (0..1).each{|x| put 't1', "#{x}", 'f1:t', 'dd'}
> hbase(main):030:0> (0..1000).each{|x| split 't1', "#{x*10}"}
> {code}
> Running overlapping splits in parallel (e.g. "#{x*10+1}", "#{x*10+2}"... ) 
> will reproduce the issue almost instantly and consistently. 
> {code}
> 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: 
> Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in 
> META
> 2012-03-28 10:57:16,321 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for 
> t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1..  
> compaction_queue=(0:1), split_queue=10
> 2012-03-28 10:57:16,343 INFO 
> org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup 
> of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; 
> Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> java.io.IOException: Failed 
> ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451)
> at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException: File does not exist: 
> /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1008)
> at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:65)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548)
> at 
> org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284)
> at org.apache.hadoop.hbase.regionserver.Store.(Store.java:221)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484)
> ... 1 more
> 2012-03-28 10:57:16,345 FATAL 
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server 
> ld2,60020,1332957343833: Abort; we got an error after point-of-no-return
> {code}
> http://hastebin.com/diqinibajo.avrasm
> later edit:
> (I'm using the la

[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase

[Jonathan Hsieh (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-5638:
--

   Resolution: Fixed
Fix Version/s: 0.94.1
   0.94.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks Matteo, and thanks for review stack.

> Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
> --
>
> Key: HBASE-5638
> URL: https://issues.apache.org/jira/browse/HBASE-5638
> Project: HBase
>  Issue Type: Sub-task
>  Components: zookeeper
>Affects Versions: 0.90.6, 0.92.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.94.1
>
> Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
> HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, 
> HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table

[Cosmin Lehene (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241586#comment-13241586
 ] 

Cosmin Lehene commented on HBASE-5665:
--

Indeed it seems to be a problem with forced splits. I'm not sure though if the 
natural splits are safe - they seem to be, but I need to test that too. 

RegionSplitPolicy.getSplitPoint() calls Store.getSplitPoint()
Store.getSplitPoint seems to do the check. 

{code}
for (StoreFile sf : storefiles) {
if (sf.isReference()) {
  // Should already be enforced since we return false in this case
  assert false : "getSplitPoint() called on a region that can't split!";
  return null;
}
{code}

BTW, we also have Store.hasReferences()
{code}
  private boolean hasReferences(Collection files) {
if (files != null && files.size() > 0) {
  for (StoreFile hsf: files) {
if (hsf.isReference()) {
  return true;
}
  }
}
return false;
  }

{code}


However here's the code in HRegion.checkSplit()
If there's an explicit split point it won't get to do the reference check.

{code}
 public byte[] checkSplit() {
// Can't split META
if (getRegionInfo().isMetaRegion()) {
  if (shouldForceSplit()) {
LOG.warn("Cannot split meta regions in HBase 0.20 and above");
  }
  return null;
}

if (this.explicitSplitPoint != null) {
  return this.explicitSplitPoint;
}

if (!splitPolicy.shouldSplit()) {
  return null;
}

byte[] ret = splitPolicy.getSplitPoint();

if (ret != null) {
  try {
checkRow(ret, "calculated split");
  } catch (IOException e) {
LOG.error("Ignoring invalid split", e);
return null;
  }
}
return ret;
  }
{code}

Multiple return points + a ret variable - this could use some polishing too :)

I'm a bit puzzled about the natural split, because, I've seen the problem with 
a forced split from UI where I don't think we provide an explicit split point. 

Cosmin

> Repeated split causes HRegionServer failures and breaks table 
> --
>
> Key: HBASE-5665
> URL: https://issues.apache.org/jira/browse/HBASE-5665
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0, 0.92.1, 0.94.0, 0.96.0, 0.94.1
>Reporter: Cosmin Lehene
>Assignee: Cosmin Lehene
>Priority: Blocker
> Attachments: HBASE-5665-0.92.patch
>
>
> Repeated splits on large tables (2 consecutive would suffice) will 
> essentially "break" the table (and the cluster), unrecoverable.
> The regionserver doing the split dies and the master will get into an 
> infinite loop trying to assign regions that seem to have the files missing 
> from HDFS.
> The table can be disabled once. upon trying to re-enable it, it will remain 
> in an intermediary state forever.
> I was able to reproduce this on a smaller table consistently.
> {code}
> hbase(main):030:0> (0..1).each{|x| put 't1', "#{x}", 'f1:t', 'dd'}
> hbase(main):030:0> (0..1000).each{|x| split 't1', "#{x*10}"}
> {code}
> Running overlapping splits in parallel (e.g. "#{x*10+1}", "#{x*10+2}"... ) 
> will reproduce the issue almost instantly and consistently. 
> {code}
> 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: 
> Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in 
> META
> 2012-03-28 10:57:16,321 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for 
> t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1..  
> compaction_queue=(0:1), split_queue=10
> 2012-03-28 10:57:16,343 INFO 
> org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup 
> of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; 
> Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> java.io.IOException: Failed 
> ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451)
> at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException: File does not exist: 
> /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputS

[jira] [Commented] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241567#comment-13241567
 ] 

Hudson commented on HBASE-5097:
---

Integrated in HBase-0.94 #65 (See 
[https://builds.apache.org/job/HBase-0.94/65/])
HBASE-5097 RegionObserver implementation whose preScannerOpen and 
postScannerOpen Impl return null can stall the system initialization through 
NPE (Ram) (Revision 1307035)

 Result = SUCCESS
ramkrishna : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


> RegionObserver implementation whose preScannerOpen and postScannerOpen Impl 
> return null can stall the system initialization through NPE
> ---
>
> Key: HBASE-5097
> URL: https://issues.apache.org/jira/browse/HBASE-5097
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5097.patch, HBASE-5097_1.patch, HBASE-5097_2.patch
>
>
> In HRegionServer.java openScanner()
> {code}
>   r.prepareScanner(scan);
>   RegionScanner s = null;
>   if (r.getCoprocessorHost() != null) {
> s = r.getCoprocessorHost().preScannerOpen(scan);
>   }
>   if (s == null) {
> s = r.getScanner(scan);
>   }
>   if (r.getCoprocessorHost() != null) {
> s = r.getCoprocessorHost().postScannerOpen(scan, s);
>   }
> {code}
> If we dont have implemention for postScannerOpen the RegionScanner is null 
> and so throwing nullpointer 
> {code}
> java.lang.NullPointerException
>   at 
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>   at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
> {code}
> Making this defect as blocker.. Pls feel free to change the priority if am 
> wrong.  Also correct me if my way of trying out coprocessors without 
> implementing postScannerOpen is wrong.  Am just a learner.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241565#comment-13241565
 ] 

Jonathan Hsieh commented on HBASE-5638:
---

Lgtm.  I'll do a quick test and commit to 0.90/0.92/0.94/trunk branches.

> Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
> --
>
> Key: HBASE-5638
> URL: https://issues.apache.org/jira/browse/HBASE-5638
> Project: HBase
>  Issue Type: Sub-task
>  Components: zookeeper
>Affects Versions: 0.90.6, 0.92.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.90.7, 0.92.2
>
> Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, 
> HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, 
> HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241560#comment-13241560
 ] 

stack commented on HBASE-5619:
--

Can you fix this Jimmy?  Happens when I try to build after applying the patch:

{code}
main:
 [exec] target/compile-proto.sh: line 12: protoc: command not found
 [exec] target/compile-proto.sh: line 12: protoc: command not found
 [exec] target/compile-proto.sh: line 12: protoc: command not found
[INFO] 
[ERROR] BUILD ERROR
[INFO] 
[INFO] An Ant BuildException has occured: exec returned: 127
{code}

> Create PB protocols for HRegionInterface
> 
>
> Key: HBASE-5619
> URL: https://issues.apache.org/jira/browse/HBASE-5619
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
> Attachments: hbase-5619.patch, hbase-5619_v3.patch, 
> hbase-5619_v4.patch
>
>
> Subtask of HBase-5443, separate HRegionInterface into admin protocol and 
> client protocol, create the PB protocol buffer files

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241563#comment-13241563
 ] 

stack commented on HBASE-5547:
--

What Lars said.

> Don't delete HFiles when in "backup mode"
> -
>
> Key: HBASE-5547
> URL: https://issues.apache.org/jira/browse/HBASE-5547
> Project: HBase
>  Issue Type: New Feature
>Reporter: Lars Hofhansl
>Assignee: Jesse Yates
>
> This came up in a discussion I had with Stack.
> It would be nice if HBase could be notified that a backup is in progress (via 
> a znode for example) and in that case either:
> 1. rename HFiles to be delete to .bck
> 2. rename the HFiles into a special directory
> 3. rename them to a general trash directory (which would not need to be tied 
> to backup mode).
> That way it should be able to get a consistent backup based on HFiles (HDFS 
> snapshots or hard links would be better options here, but we do not have 
> those).
> #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241556#comment-13241556
 ] 

stack commented on HBASE-5665:
--

Or, is this a problem only with forced splits?  It doesn't happen when we split 
'naturally' because we'll check for references?

> Repeated split causes HRegionServer failures and breaks table 
> --
>
> Key: HBASE-5665
> URL: https://issues.apache.org/jira/browse/HBASE-5665
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0, 0.92.1, 0.94.0, 0.96.0, 0.94.1
>Reporter: Cosmin Lehene
>Assignee: Cosmin Lehene
>Priority: Blocker
> Attachments: HBASE-5665-0.92.patch
>
>
> Repeated splits on large tables (2 consecutive would suffice) will 
> essentially "break" the table (and the cluster), unrecoverable.
> The regionserver doing the split dies and the master will get into an 
> infinite loop trying to assign regions that seem to have the files missing 
> from HDFS.
> The table can be disabled once. upon trying to re-enable it, it will remain 
> in an intermediary state forever.
> I was able to reproduce this on a smaller table consistently.
> {code}
> hbase(main):030:0> (0..1).each{|x| put 't1', "#{x}", 'f1:t', 'dd'}
> hbase(main):030:0> (0..1000).each{|x| split 't1', "#{x*10}"}
> {code}
> Running overlapping splits in parallel (e.g. "#{x*10+1}", "#{x*10+2}"... ) 
> will reproduce the issue almost instantly and consistently. 
> {code}
> 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: 
> Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in 
> META
> 2012-03-28 10:57:16,321 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for 
> t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1..  
> compaction_queue=(0:1), split_queue=10
> 2012-03-28 10:57:16,343 INFO 
> org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup 
> of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; 
> Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> java.io.IOException: Failed 
> ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451)
> at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException: File does not exist: 
> /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1008)
> at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:65)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548)
> at 
> org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284)
> at org.apache.hadoop.hbase.regionserver.Store.(Store.java:221)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484)
> ... 1 more
> 2012-03-28 10:57:16,345 FATAL 
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server 
> ld2,60020,1332957343833: Abort; we got an error after point-of-no-return
> {code}
> http://hastebin.com/diqinibajo.avrasm
> 

[jira] [Commented] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241555#comment-13241555
 ] 

stack commented on HBASE-5665:
--

Did we drop the check for references along the way Cosmin?  It used to be 
impossible to even attempt a split of a region with references.  Are you 
working on it?  I agree this a blocker all around.

> Repeated split causes HRegionServer failures and breaks table 
> --
>
> Key: HBASE-5665
> URL: https://issues.apache.org/jira/browse/HBASE-5665
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0, 0.92.1, 0.94.0, 0.96.0, 0.94.1
>Reporter: Cosmin Lehene
>Assignee: Cosmin Lehene
>Priority: Blocker
> Attachments: HBASE-5665-0.92.patch
>
>
> Repeated splits on large tables (2 consecutive would suffice) will 
> essentially "break" the table (and the cluster), unrecoverable.
> The regionserver doing the split dies and the master will get into an 
> infinite loop trying to assign regions that seem to have the files missing 
> from HDFS.
> The table can be disabled once. upon trying to re-enable it, it will remain 
> in an intermediary state forever.
> I was able to reproduce this on a smaller table consistently.
> {code}
> hbase(main):030:0> (0..1).each{|x| put 't1', "#{x}", 'f1:t', 'dd'}
> hbase(main):030:0> (0..1000).each{|x| split 't1', "#{x*10}"}
> {code}
> Running overlapping splits in parallel (e.g. "#{x*10+1}", "#{x*10+2}"... ) 
> will reproduce the issue almost instantly and consistently. 
> {code}
> 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: 
> Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in 
> META
> 2012-03-28 10:57:16,321 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for 
> t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1..  
> compaction_queue=(0:1), split_queue=10
> 2012-03-28 10:57:16,343 INFO 
> org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup 
> of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; 
> Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> java.io.IOException: Failed 
> ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451)
> at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException: File does not exist: 
> /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1008)
> at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:65)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548)
> at 
> org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284)
> at org.apache.hadoop.hbase.regionserver.Store.(Store.java:221)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504)
> at 
> org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484)
> ... 1 more
> 2012-03-28 10:57:16,345 FATAL 
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server 
> ld2,60020,1332957343833: Abort; we got an error after