date:20120202


[ 
https://issues.apache.org/jira/browse/HBASE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198635#comment-13198635
 ] 

Phabricator commented on HBASE-4542:


zhiqiu has commented on the revision [jira] [HBASE-4542] Add filter info to 
slow query logging.

  @stack I just attached a patch to JIRA 
(https://issues.apache.org/jira/browse/HBASE-4542). Thanks!

REVISION DETAIL
  https://reviews.facebook.net/D1539


 add filter info to slow query logging
 -

 Key: HBASE-4542
 URL: https://issues.apache.org/jira/browse/HBASE-4542
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.89.20100924
Reporter: Kannan Muthukkaruppan
Assignee: Madhuwanti Vaidya
 Attachments: 
 0001-jira-HBASE-4542-Add-filter-info-to-slow-query-loggin.patch, 
 D1263.2.patch, D1539.1.patch


 Slow query log doesn't report filters in effect.
 For example:
 {code}
 (operationTooSlow): \
 {processingtimems:3468,client:10.138.43.206:40035,timeRange: 
 [0,9223372036854775807],\
 starttimems:1317772005821,responsesize:42411, \
 class:HRegionServer,table:myTable,families:{CF1:ALL]},\
 row:6c3b8efa132f0219b7621ed1e5c8c70b,queuetimems:0,\
 method:get,totalColumns:1,maxVersions:1,storeLimit:-1}
 {code}
 the above would suggest that all columns of myTable:CF1 are being requested 
 for the given row. But in reality there could be filters in effect (such as 
 ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should 
 enhance the slow query log to capture  report this information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5212) Fix test TestTableMapReduce against 0.23.


[ 
https://issues.apache.org/jira/browse/HBASE-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198654#comment-13198654
 ] 

Hudson commented on HBASE-5212:
---

Integrated in HBase-0.92 #270 (See 
[https://builds.apache.org/job/HBase-0.92/270/])
HBASE-5212 Fix test TestTableMapReduce against 0.23 (Ted and Gregory)

jmhsieh : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/pom.xml
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java


 Fix test TestTableMapReduce against 0.23.
 -

 Key: HBASE-5212
 URL: https://issues.apache.org/jira/browse/HBASE-5212
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Mahadev konar
Assignee: Gregory Chanan
 Fix For: 0.94.0, 0.92.1

 Attachments: 5212-v2.txt, HBASE-5212-v3.patch, HBASE-5212.patch


 As reported by Andrew on the hadoop mailing list, mvn -Dhadoop.profile=23 
 clean test -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce fails 
 on 0.92 branch. There are minor changes to HBase poms required to fix that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4542) add filter info to slow query logging

2012-02-02 Thread ramkrishna.s.vasudevan (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198669#comment-13198669
]

Hadoop QA commented on HBASE-4542:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12512929/0001-jira-HBASE-4542-Add-filter-info-to-slow-query-loggin.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 5 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated -140 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 157 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.io.hfile.TestHFileBlock
org.apache.hadoop.hbase.mapreduce.TestImportTsv

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/894//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/894//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/894//console

This message is automatically generated.

add filter info to slow query logging
-

Key: HBASE-4542
URL: https://issues.apache.org/jira/browse/HBASE-4542
Project: HBase
Issue Type: Improvement
Affects Versions: 0.89.20100924
Reporter: Kannan Muthukkaruppan
Assignee: Madhuwanti Vaidya
Attachments:
0001-jira-HBASE-4542-Add-filter-info-to-slow-query-loggin.patch,
D1263.2.patch, D1539.1.patch

Slow query log doesn't report filters in effect.
For example:
{code}
(operationTooSlow): \
{processingtimems:3468,client:10.138.43.206:40035,timeRange:
[0,9223372036854775807],\
starttimems:1317772005821,responsesize:42411, \
class:HRegionServer,table:myTable,families:{CF1:ALL]},\
row:6c3b8efa132f0219b7621ed1e5c8c70b,queuetimems:0,\
method:get,totalColumns:1,maxVersions:1,storeLimit:-1}
{code}
the above would suggest that all columns of myTable:CF1 are being requested
for the given row. But in reality there could be filters in effect (such as
ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should
enhance the slow query log to capture report this information.

[jira] [Commented] (HBASE-5314) Gracefully rolling restart region servers in rolling-restart.sh

2012-02-02 Thread YiFeng Jiang (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198799#comment-13198799
 ] 

YiFeng Jiang commented on HBASE-5314:
-

@stack
I got it. I will add a --graceful option that enables this new facility 
(disabled by default).

 Gracefully rolling restart region servers in rolling-restart.sh
 ---

 Key: HBASE-5314
 URL: https://issues.apache.org/jira/browse/HBASE-5314
 Project: HBase
  Issue Type: Improvement
  Components: scripts
Reporter: YiFeng Jiang
Priority: Minor
 Attachments: HBASE-5314.patch


 The rolling-restart.sh has a --rs-only option which simply restarts all 
 region servers in the cluster.
 Consider improve it to gracefully restart region servers to avoid the offline 
 time of the regions deployed on that server, and keep the region 
 distributions same as what it was before the restarting.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5314) Gracefully rolling restart region servers in rolling-restart.sh

2012-02-02 Thread YiFeng Jiang (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

YiFeng Jiang updated HBASE-5314:


Attachment: HBASE-5314.patch.2

A new patch adding a --graceful option to enable the new gracefully rolling 
restart facility.

 Gracefully rolling restart region servers in rolling-restart.sh
 ---

 Key: HBASE-5314
 URL: https://issues.apache.org/jira/browse/HBASE-5314
 Project: HBase
  Issue Type: Improvement
  Components: scripts
Reporter: YiFeng Jiang
Priority: Minor
 Attachments: HBASE-5314.patch, HBASE-5314.patch.2


 The rolling-restart.sh has a --rs-only option which simply restarts all 
 region servers in the cluster.
 Consider improve it to gracefully restart region servers to avoid the offline 
 time of the regions deployed on that server, and keep the region 
 distributions same as what it was before the restarting.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5314) Gracefully rolling restart region servers in rolling-restart.sh

2012-02-02 Thread YiFeng Jiang (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198824#comment-13198824
 ] 

YiFeng Jiang commented on HBASE-5314:
-

I am using HBase 0.92.0.

 Gracefully rolling restart region servers in rolling-restart.sh
 ---

 Key: HBASE-5314
 URL: https://issues.apache.org/jira/browse/HBASE-5314
 Project: HBase
  Issue Type: Improvement
  Components: scripts
Reporter: YiFeng Jiang
Priority: Minor
 Attachments: HBASE-5314.patch, HBASE-5314.patch.2


 The rolling-restart.sh has a --rs-only option which simply restarts all 
 region servers in the cluster.
 Consider improve it to gracefully restart region servers to avoid the offline 
 time of the regions deployed on that server, and keep the region 
 distributions same as what it was before the restarting.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5321) this.allRegionServersOffline not set to false after one RS comes online and assignment is done in 0.90.


 [ 
https://issues.apache.org/jira/browse/HBASE-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-5321:
--

Attachment: HBASE-5321.patch

Please review the patch.

 this.allRegionServersOffline  not set to false after one RS comes online and 
 assignment is done in 0.90.
 

 Key: HBASE-5321
 URL: https://issues.apache.org/jira/browse/HBASE-5321
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.90.6

 Attachments: HBASE-5321.patch


 In HBASE-5160 we do not wait for TM to assign the regions after the first RS 
 comes online.
 After doing this the variable this.allRegionServersOffline needs to be reset 
 which is not done in 0.90.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5186) Add metrics to ThriftServer

2012-02-02 Thread Zhihong Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5186:
--

Attachment: 5186-v12.txt

Patch v12 adds logging to CallQueue.java

 Add metrics to ThriftServer
 ---

 Key: HBASE-5186
 URL: https://issues.apache.org/jira/browse/HBASE-5186
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Attachments: 5186-v10.txt, 5186-v11.txt, 5186-v12.txt, 5186-v9.txt, 
 HBASE-5186.D1461.1.patch, HBASE-5186.D1461.2.patch, HBASE-5186.D1461.3.patch, 
 HBASE-5186.D1461.4.patch, HBASE-5186.D1461.5.patch, HBASE-5186.D1461.6.patch, 
 HBASE-5186.D1461.7.patch, HBASE-5186.D1461.8.patch


 It will be useful to have some metrics (queue length, waiting time, 
 processing time ...) similar to Hadoop RPC server. This allows us to monitor 
 system health also provide a tool to diagnose the problem where thrift calls 
 are slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5322) CLONE - getScanner hangs with some startRows that are found if scanning entire table

2012-02-02 Thread Karthik Pandian (Created) (JIRA)

CLONE - getScanner hangs with some startRows that are found if scanning entire 
table


 Key: HBASE-5322
 URL: https://issues.apache.org/jira/browse/HBASE-5322
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.18.0, 0.18.1, 0.19.0
Reporter: Karthik Pandian
Priority: Critical
 Fix For: 0.18.1, 0.19.0


I have a table with 8 byte binary row keys.  There are a a few hundred 
thousands rows, each with two families and between 1k and 50k of total data 
across about 15 columns.

When attempting to get a scanner using a specified startRow, my client freezes 
on the HT.getScanner(cols,row) with no exception ever thrown and no debug 
output in any server logs.

If I get a scanner with HT.getScanner(cols) and then iterate through, I will 
eventually reach the row I was seeking before successfully.

Some rows can be found, some cannot.  At this point I'm not able to distinguish 
anything special about the ones that cause the client the hang.

At first I thought this was only a problem with 0.19 trunk as a downgrade to 
0.18 resolved the issue for a particular key.  However other keys still have 
this issue on 0.18 branch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5322) CLONE - getScanner hangs with some startRows that are found if scanning entire table

2012-02-02 Thread Karthik Pandian (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Karthik Pandian updated HBASE-5322:
---

Description:
I have a hbase table which holds data for more than 10GB. Now I used the same
client scanner to scan which fails and reports,

Could not seek StoreFileScanner[HFileScanner for reader reader=hdfs.

This issue occurs only for the table which holds huge data and not for tables
holding small data.

was:
I have a table with 8 byte binary row keys. There are a a few hundred
thousands rows, each with two families and between 1k and 50k of total data
across about 15 columns.

When attempting to get a scanner using a specified startRow, my client freezes
on the HT.getScanner(cols,row) with no exception ever thrown and no debug
output in any server logs.

If I get a scanner with HT.getScanner(cols) and then iterate through, I will
eventually reach the row I was seeking before successfully.

Some rows can be found, some cannot. At this point I'm not able to distinguish
anything special about the ones that cause the client the hang.

At first I thought this was only a problem with 0.19 trunk as a downgrade to
0.18 resolved the issue for a particular key. However other keys still have
this issue on 0.18 branch.

Priority: Blocker (was: Critical)
Affects Version/s: (was: 0.18.1)
(was: 0.19.0)
(was: 0.18.0)
0.90.4
Fix Version/s: (was: 0.18.1)
(was: 0.19.0)

CLONE - getScanner hangs with some startRows that are found if scanning
entire table

Key: HBASE-5322
URL: https://issues.apache.org/jira/browse/HBASE-5322
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Karthik Pandian
Priority: Blocker

I have a hbase table which holds data for more than 10GB. Now I used the same
client scanner to scan which fails and reports,
Could not seek StoreFileScanner[HFileScanner for reader reader=hdfs.
This issue occurs only for the table which holds huge data and not for tables
holding small data.

[jira] [Commented] (HBASE-5186) Add metrics to ThriftServer

[
https://issues.apache.org/jira/browse/HBASE-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198892#comment-13198892
]

Hadoop QA commented on HBASE-5186:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512964/5186-v12.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 5 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated -140 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 157 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/895//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/895//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/895//console

This message is automatically generated.

Add metrics to ThriftServer
---

Key: HBASE-5186
URL: https://issues.apache.org/jira/browse/HBASE-5186
Project: HBase
Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
Attachments: 5186-v10.txt, 5186-v11.txt, 5186-v12.txt, 5186-v9.txt,
HBASE-5186.D1461.1.patch, HBASE-5186.D1461.2.patch, HBASE-5186.D1461.3.patch,
HBASE-5186.D1461.4.patch, HBASE-5186.D1461.5.patch, HBASE-5186.D1461.6.patch,
HBASE-5186.D1461.7.patch, HBASE-5186.D1461.8.patch

It will be useful to have some metrics (queue length, waiting time,
processing time ...) similar to Hadoop RPC server. This allows us to monitor
system health also provide a tool to diagnose the problem where thrift calls
are slow.

[jira] [Commented] (HBASE-5321) this.allRegionServersOffline not set to false after one RS comes online and assignment is done in 0.90.


[ 
https://issues.apache.org/jira/browse/HBASE-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198996#comment-13198996
 ] 

Zhihong Yu commented on HBASE-5321:
---

Patch looks good.

 this.allRegionServersOffline  not set to false after one RS comes online and 
 assignment is done in 0.90.
 

 Key: HBASE-5321
 URL: https://issues.apache.org/jira/browse/HBASE-5321
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.90.6

 Attachments: HBASE-5321.patch


 In HBASE-5160 we do not wait for TM to assign the regions after the first RS 
 comes online.
 After doing this the variable this.allRegionServersOffline needs to be reset 
 which is not done in 0.90.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5186) Add metrics to ThriftServer

2012-02-02 Thread Zhihong Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5186:
--

Fix Version/s: 0.94.0
 Hadoop Flags: Reviewed

 Add metrics to ThriftServer
 ---

 Key: HBASE-5186
 URL: https://issues.apache.org/jira/browse/HBASE-5186
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.94.0

 Attachments: 5186-v10.txt, 5186-v11.txt, 5186-v12.txt, 5186-v9.txt, 
 HBASE-5186.D1461.1.patch, HBASE-5186.D1461.2.patch, HBASE-5186.D1461.3.patch, 
 HBASE-5186.D1461.4.patch, HBASE-5186.D1461.5.patch, HBASE-5186.D1461.6.patch, 
 HBASE-5186.D1461.7.patch, HBASE-5186.D1461.8.patch


 It will be useful to have some metrics (queue length, waiting time, 
 processing time ...) similar to Hadoop RPC server. This allows us to monitor 
 system health also provide a tool to diagnose the problem where thrift calls 
 are slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5186) Add metrics to ThriftServer


[ 
https://issues.apache.org/jira/browse/HBASE-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199015#comment-13199015
 ] 

Zhihong Yu commented on HBASE-5186:
---

Integrated to TRUNK.

Thanks for the patch Scott.

 Add metrics to ThriftServer
 ---

 Key: HBASE-5186
 URL: https://issues.apache.org/jira/browse/HBASE-5186
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.94.0

 Attachments: 5186-v10.txt, 5186-v11.txt, 5186-v12.txt, 5186-v9.txt, 
 HBASE-5186.D1461.1.patch, HBASE-5186.D1461.2.patch, HBASE-5186.D1461.3.patch, 
 HBASE-5186.D1461.4.patch, HBASE-5186.D1461.5.patch, HBASE-5186.D1461.6.patch, 
 HBASE-5186.D1461.7.patch, HBASE-5186.D1461.8.patch


 It will be useful to have some metrics (queue length, waiting time, 
 processing time ...) similar to Hadoop RPC server. This allows us to monitor 
 system health also provide a tool to diagnose the problem where thrift calls 
 are slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199018#comment-13199018
 ] 

Jesse Yates commented on HBASE-5304:


+1 ship it.

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4658) Put attributes are not exposed via the ThriftServer


[ 
https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199020#comment-13199020
 ] 

Phabricator commented on HBASE-4658:


tedyu has commented on the revision [jira] [HBASE-4658] Put attributes are not 
exposed via the ThriftServer.

INLINE COMMENTS
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:497 It would 
be better to replace Put with 'mutation'
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:520 It would 
be better to replace Put with 'mutation'
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:537 It would 
be better to replace Put with 'mutation'
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:557 It would 
be better to replace Put with 'mutation'

REVISION DETAIL
  https://reviews.facebook.net/D1563


 Put attributes are not exposed via the ThriftServer
 ---

 Key: HBASE-4658
 URL: https://issues.apache.org/jira/browse/HBASE-4658
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1563.1.patch, D1563.1.patch, D1563.1.patch, 
 D1563.2.patch, D1563.2.patch, D1563.2.patch, ThriftPutAttributes1.txt


 The Put api also takes in a bunch of arbitrary attributes that an application 
 can use to associate metadata with each put operation. This is not exposed 
 via Thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4658) Put attributes are not exposed via the ThriftServer


[ 
https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199022#comment-13199022
 ] 

Phabricator commented on HBASE-4658:


tedyu has commented on the revision [jira] [HBASE-4658] Put attributes are not 
exposed via the ThriftServer.

INLINE COMMENTS
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:497 It would 
be better to replace Put with 'mutation'
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:520 It would 
be better to replace Put with 'mutation'
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:537 It would 
be better to replace Put with 'mutation'
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:557 It would 
be better to replace Put with 'mutation'

REVISION DETAIL
  https://reviews.facebook.net/D1563


 Put attributes are not exposed via the ThriftServer
 ---

 Key: HBASE-4658
 URL: https://issues.apache.org/jira/browse/HBASE-4658
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1563.1.patch, D1563.1.patch, D1563.1.patch, 
 D1563.2.patch, D1563.2.patch, D1563.2.patch, ThriftPutAttributes1.txt


 The Put api also takes in a bunch of arbitrary attributes that an application 
 can use to associate metadata with each put operation. This is not exposed 
 via Thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4658) Put attributes are not exposed via the ThriftServer

2012-02-02 Thread ramkrishna.s.vasudevan (Created) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199023#comment-13199023
 ] 

Phabricator commented on HBASE-4658:


tedyu has commented on the revision [jira] [HBASE-4658] Put attributes are not 
exposed via the ThriftServer.

INLINE COMMENTS
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:497 It would 
be better to replace Put with 'mutation'
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:520 It would 
be better to replace Put with 'mutation'
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:537 It would 
be better to replace Put with 'mutation'
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift:557 It would 
be better to replace Put with 'mutation'

REVISION DETAIL
  https://reviews.facebook.net/D1563


 Put attributes are not exposed via the ThriftServer
 ---

 Key: HBASE-4658
 URL: https://issues.apache.org/jira/browse/HBASE-4658
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1563.1.patch, D1563.1.patch, D1563.1.patch, 
 D1563.2.patch, D1563.2.patch, D1563.2.patch, ThriftPutAttributes1.txt


 The Put api also takes in a bunch of arbitrary attributes that an application 
 can use to associate metadata with each put operation. This is not exposed 
 via Thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5323) Need to handle assertion error if split log through ServerShutDownHandler by shutting down the master

Need to handle assertion error if split log through ServerShutDownHandler by 
shutting down the master
-

 Key: HBASE-5323
 URL: https://issues.apache.org/jira/browse/HBASE-5323
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
 Fix For: 0.94.0, 0.90.7


We know that while parsing the HLog we expect the proper length from HDFS.
In WALReaderFSDataInputStream
{code}
  assert(realLength = this.length);
{code}
We are trying to come out if the above condition is not satisfied.  But if 
SSH.splitLog() gets this problem then it lands in the run method of 
EventHandler.  This kills the SSH thread and so further assignment does not 
happen.  If ROOT and META are to be assigned they cannot be.
I think in this condition we abort the master by catching such exceptions.
Please do suggest.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5323) Need to handle assertion error while splitting log through ServerShutDownHandler by shutting down the master

2012-02-02 Thread ramkrishna.s.vasudevan (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

ramkrishna.s.vasudevan updated HBASE-5323:
--

Summary: Need to handle assertion error while splitting log through
ServerShutDownHandler by shutting down the master (was: Need to handle
assertion error if split log through ServerShutDownHandler by shutting down the
master)

Need to handle assertion error while splitting log through
ServerShutDownHandler by shutting down the master

Key: HBASE-5323
URL: https://issues.apache.org/jira/browse/HBASE-5323
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
Fix For: 0.94.0, 0.90.7

We know that while parsing the HLog we expect the proper length from HDFS.
In WALReaderFSDataInputStream
{code}
assert(realLength = this.length);
{code}
We are trying to come out if the above condition is not satisfied. But if
SSH.splitLog() gets this problem then it lands in the run method of
EventHandler. This kills the SSH thread and so further assignment does not
happen. If ROOT and META are to be assigned they cannot be.
I think in this condition we abort the master by catching such exceptions.
Please do suggest.

[jira] [Commented] (HBASE-5323) Need to handle assertion error while splitting log through ServerShutDownHandler by shutting down the master

2012-02-02 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199029#comment-13199029
 ] 

Zhihong Yu commented on HBASE-5323:
---

Suggested approach makes sense.

 Need to handle assertion error while splitting log through 
 ServerShutDownHandler by shutting down the master
 

 Key: HBASE-5323
 URL: https://issues.apache.org/jira/browse/HBASE-5323
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
 Fix For: 0.94.0, 0.90.7


 We know that while parsing the HLog we expect the proper length from HDFS.
 In WALReaderFSDataInputStream
 {code}
   assert(realLength = this.length);
 {code}
 We are trying to come out if the above condition is not satisfied.  But if 
 SSH.splitLog() gets this problem then it lands in the run method of 
 EventHandler.  This kills the SSH thread and so further assignment does not 
 happen.  If ROOT and META are to be assigned they cannot be.
 I think in this condition we abort the master by catching such exceptions.
 Please do suggest.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4177) Handling read failures during recovery‏ - when HMaster calls Namenode recovery, recovery may be a failure leading to read failure while splitting logs

[
https://issues.apache.org/jira/browse/HBASE-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199033#comment-13199033
]

ramkrishna.s.vasudevan commented on HBASE-4177:
---

Any suggestions on this. We tend to run into this problem every now and then.

Handling read failures during recovery‏ - when HMaster calls Namenode
recovery, recovery may be a failure leading to read failure while splitting
logs
--

Key: HBASE-4177
URL: https://issues.apache.org/jira/browse/HBASE-4177
Project: HBase
Issue Type: Bug
Components: master
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical

As per the mailing thread with the heading
'Handling read failures during recovery‏' we found this problem.
As part of split Logs the HMaster calls Namenode recovery. The recovery is
an asynchronous process.
In HDFS
===
Even though client is getting the updated block info from Namenode on first
read failure, client is discarding the new info and using the old info only
to retrieve the data from datanode. So, all the read
retries are failing. [Method parameter reassignment - Not reflected in
caller].
In HBASE
===
In HMaster code we tend to wait for 1sec. But if the recovery had some
failure then split log may not happen and may lead to dataloss.
So may be we need to decide upon the actual delay that needs to be introduced
once Hmaster calls NN recovery.

[jira] [Updated] (HBASE-5299) CatalogTracker.getMetaServerConnection() checks for root server connection and makes waitForMeta to go into infinite wait in region assignment flow.

2012-02-02 Thread ramkrishna.s.vasudevan (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-5299:
--

Summary: CatalogTracker.getMetaServerConnection() checks for root server 
connection and makes waitForMeta to go into infinite wait in region assignment 
flow.  (was: CatalogTracker.getMetaServerConnection() checks for root server 
connection and makes waitForMeta to go into infinite loop in region assignment 
flow.)

 CatalogTracker.getMetaServerConnection() checks for root server connection 
 and makes waitForMeta to go into infinite wait in region assignment flow.
 

 Key: HBASE-5299
 URL: https://issues.apache.org/jira/browse/HBASE-5299
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor

 RSA, RS B and RS C are 3 region servers.
 RS A - META
 RS B - ROOT
 RS C - NON META and NON ROOT
 Kill RS B and wait for server shutdown handler to start.  
 Start RS B again before assigning ROOT to RS C.
 Now the cluster will try to assign new regions to RS B.  
 But as ROOT is not yet assigned the OpenRegionHandler.updateMeta will fail to 
 update the regions just because ROOT is not online.
 {code}
 a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENING
 2012-01-30 16:23:25,126 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 regionserver:60020-0x1352e27539c0009 Attempting to transition node 
 a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENING
 2012-01-30 16:23:25,159 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 regionserver:60020-0x1352e27539c0009 Successfully transitioned node 
 a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENING
 2012-01-30 16:23:35,385 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 regionserver:60020-0x1352e27539c0009 Attempting to transition node 
 a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENING
 2012-01-30 16:23:35,449 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 regionserver:60020-0x1352e27539c0009 Successfully transitioned node 
 a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENING
 2012-01-30 16:24:16,666 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 regionserver:60020-0x1352e27539c0009 Attempting to transition node 
 a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENING
 2012-01-30 16:24:16,701 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
 regionserver:60020-0x1352e27539c0009 Successfully transitioned node 
 a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to 
 RS_ZK_REGION_OPENING
 2012-01-30 16:24:20,788 DEBUG 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Interrupting 
 thread Thread[PostOpenDeployTasks:a87109263ed53e67158377a149c5a7be,5,main]
 2012-01-30 16:24:30,699 WARN 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception 
 running postOpenDeployTasks; region=a87109263ed53e67158377a149c5a7be
 org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Interrupted
   at 
 org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:439)
   at 
 org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1382)
   at 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:221)
 {code}
 So we need to wait for TM to assign the regions again. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199046#comment-13199046
 ] 

Zhihong Yu commented on HBASE-5304:
---

Wait, TestInstantSchemaChangeSplit was a new test failure.

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache

[
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199072#comment-13199072
]

Phabricator commented on HBASE-5074:

dhruba has commented on the revision [jira] [HBASE-5074] Support checksums in
HBase block cache.

INLINE COMMENTS

src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:206-207
will fix
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:1072 sure
src/main/java/org/apache/hadoop/hbase/HConstants.java:598 I will make this
part of the code cleaner. I still am hoping to keep only one knob: whether to
verify hbase checksums or not. If hbase checksums is switched on, then hdfs
checksums will automatically be switched off. If hbase checksums is configured
'off', then it will automatically switch on hdfs checksums. I feel that the
other knobs (e.g. no checksums at all or use both checksums) are not very
interesting in *any* production environment and I would like to keep the code
complexity a little lower by avoiding those two combinations. Hope that is ok
with you.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:3597 Good
idea, will do

src/main/java/org/apache/hadoop/hbase/util/ChecksumByteArrayOutputStream.java:31
It tried this, but it needs a few changes, so I anyway landed up with needing
my own object wrapper over DataOutputBuffer.
src/main/java/org/apache/hadoop/hbase/util/ChecksumFactory.java:39 I too feel
that we should add the checksum type to the hfileblock header. That will make
us future proof to try new checksum algorithms in the future. Will make this
change.
src/main/java/org/apache/hadoop/hbase/util/HFileSystem.java:132-133 This is
equivalent to the existing FileSystem.get() and many places in hbase uses this.
src/main/java/org/apache/hadoop/hbase/util/HFileSystem.java:80 I will make
this public so that users can create a HFileSystem object on a non-default path
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:102 I am
making changes here based on mikhial's suggestion too.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:229 as you
would see, the existing code path that create a HFileBlock usin g this
constructor uses it for only in-memory caching, so it never fills up or uses
the onDiskDataSizeWithHeader field. But I will set it to what you propose.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:252
ondisksizewithheader = ondiskdatasizewithheader + checksum bytes
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:751 I am in
complete agreement with you. I wish I could have used the hadoop trunk code
here. Unfortunately, hbase pulls in hadoop 1.0 which does not have this
implementation. Another option is to make a copy of this code from hadoop into
hbase code, but this has its own set of problems for maintainability. I am
hoping that hbase will move to hadoop 2.0 very soon and then we can start the
more optimal checksum implementation. Hope that is ok with you.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:1401-1402 This
needs to be thread safe.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:1634 This is
an internal method and this error is handled by upper layers (by switching off
hbase checksums). So, I am following the paradigm of using Exceptions only when
true errors happen; I would like to avoid writing code that generates
exceptions in one layer catches them in another layer and handles them. The
discussion with Doug Cutting on the hdfs-symlink patch is etched in my mind:-)
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:1888 I will
work (in a later patch) to use bulk checksum verifications, using native code,
etc (from hadoop) in a later patch. I would like to keep this patch smaller
that what it already is by focussing on the disk format change, compatibility
with older versions, etc. The main reason is that most of the hadoop checksum
optimizations are only in hadoop 2.0. I am hoping that it is ok with you.

REVISION DETAIL
https://reviews.facebook.net/D1521

support checksums in HBase block cache
--

Key: HBASE-5074
URL: https://issues.apache.org/jira/browse/HBASE-5074
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Attachments: D1521.1.patch, D1521.1.patch

The current implementation of HDFS stores the data in one block file and the
metadata(checksum) in another block file. This means that every read into the
HBase block cache actually consumes two disk iops, one to the datafile and
one to the checksum file. This is a major problem for scaling HBase, because
HBase is usually bottlenecked

[jira] [Commented] (HBASE-5318) Support Eclipse Indigo

[
https://issues.apache.org/jira/browse/HBASE-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199071#comment-13199071
]

Jesse Yates commented on HBASE-5318:

Support Eclipse Indigo
---

Key: HBASE-5318
URL: https://issues.apache.org/jira/browse/HBASE-5318
Project: HBase
Issue Type: Improvement
Components: build
Affects Versions: 0.94.0
Environment: Eclipse Indigo (1.4.1) which includes m2eclipse (1.0
SR1).
Reporter: Jesse Yates
Assignee: Jesse Yates
Priority: Minor
Labels: maven
Attachments: mvn_HBASE-5318_r0.patch

The current 'standard' release of Eclipse (indigo) comes with m2eclipse
installed. However, as of m2e v1.0, interesting lifecycle phases are now
handled via a 'connector'. However, several of the plugins we use don't
support connectors. This means that eclipse bails out and won't build the
project or view it as 'working' even though it builds just fine from the the
command line.
Since Eclipse is one of the major java IDEs and that Indigo has been around
for a while, we should make it easy to for new devs to pick up the code and
for older devs to upgrade painlessly. The original build should not be
modified in any significant way.

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199070#comment-13199070
 ] 

Lars Hofhansl commented on HBASE-5304:
--

I noticed... Will check it out of course.

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache

[
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199073#comment-13199073
]

Phabricator commented on HBASE-5074:

dhruba has commented on the revision [jira] [HBASE-5074] Support checksums in
HBase block cache.

INLINE COMMENTS

REVISION DETAIL
https://reviews.facebook.net/D1521

support checksums in HBase block cache
--

[jira] [Updated] (HBASE-5074) support checksums in HBase block cache


 [ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5074:
---

Attachment: D1521.2.patch

dhruba updated the revision [jira] [HBASE-5074] Support checksums in HBase 
block cache.
Reviewers: mbautin

  Addressed first-level comments from Todd and Mikhail.
  All awesome feedback, thanks a lot folks!

  There are three main things that are not in this patch yet:
  make bytesPerChecksum configurable, add 'checksum type' to the header,
  and work on making AbstractFSReader.getStream()
  thread safe. I will post these three fixes in a day or so.

REVISION DETAIL
  https://reviews.facebook.net/D1521

AFFECTED FILES
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
  src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
  
src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
  
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
  src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
  src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
  src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
  src/main/java/org/apache/hadoop/hbase/HConstants.java
  src/main/java/org/apache/hadoop/hbase/util/HFileSystem.java
  src/main/java/org/apache/hadoop/hbase/util/ChecksumByteArrayOutputStream.java
  src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java
  src/main/java/org/apache/hadoop/hbase/util/ChecksumFactory.java
  
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
  
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java


 support checksums in HBase block cache
 --

 Key: HBASE-5074
 URL: https://issues.apache.org/jira/browse/HBASE-5074
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1521.1.patch, D1521.1.patch, D1521.2.patch, 
 D1521.2.patch


 The current implementation of HDFS stores the data in one block file and the 
 metadata(checksum) in another block file. This means that every read into the 
 HBase block cache actually consumes two disk iops, one to the datafile and 
 one to the

[jira] [Updated] (HBASE-5074) support checksums in HBase block cache


 [ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5074:
---

Attachment: D1521.2.patch

dhruba updated the revision [jira] [HBASE-5074] Support checksums in HBase 
block cache.
Reviewers: mbautin

  Addressed first-level comments from Todd and Mikhail.
  All awesome feedback, thanks a lot folks!

  There are three main things that are not in this patch yet:
  make bytesPerChecksum configurable, add 'checksum type' to the header,
  and work on making AbstractFSReader.getStream()
  thread safe. I will post these three fixes in a day or so.

REVISION DETAIL
  https://reviews.facebook.net/D1521

AFFECTED FILES
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
  src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
  
src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
  
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
  src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
  src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
  src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
  src/main/java/org/apache/hadoop/hbase/HConstants.java
  src/main/java/org/apache/hadoop/hbase/util/HFileSystem.java
  src/main/java/org/apache/hadoop/hbase/util/ChecksumByteArrayOutputStream.java
  src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java
  src/main/java/org/apache/hadoop/hbase/util/ChecksumFactory.java
  
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
  
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java


 support checksums in HBase block cache
 --

 Key: HBASE-5074
 URL: https://issues.apache.org/jira/browse/HBASE-5074
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1521.1.patch, D1521.1.patch, D1521.2.patch, 
 D1521.2.patch


 The current implementation of HDFS stores the data in one block file and the 
 metadata(checksum) in another block file. This means that every read into the 
 HBase block cache actually consumes two disk iops, one to the datafile and 
 one to the

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199088#comment-13199088
 ] 

Lars Hofhansl commented on HBASE-5304:
--

TestInstantSchemaChangeSplit passes locally. Not sure what is going on with 
precommit and why we see all these spurious failures.

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199094#comment-13199094
 ] 

Jesse Yates commented on HBASE-5304:


A lot of it can be if jenkins is sketchy. Can we rerun it on jenkins to see if 
we get the failures again?

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199100#comment-13199100
 ] 

Zhihong Yu commented on HBASE-5304:
---

{code}
+if (this.region.getExplicitSplitPoint() != null) {
+  return this.region.getExplicitSplitPoint();
{code}
Can we store the return value from region.getExplicitSplitPoint() in the above 
case ?

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199109#comment-13199109
 ] 

Zhihong Yu commented on HBASE-5304:
---

Table creation may take some time.
The test failure was caused by findRSWithOnlineRegionFor() not retrying.

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199148#comment-13199148
 ] 

Lars Hofhansl commented on HBASE-5304:
--

@Ted: re: storing the value... Sure. I'll do that at commit, unless you have 
other objections.
I'll reattach the same file to another jenkins build.

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5304) Pluggable split key policy


 [ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5304:
-

Attachment: 5304-v7.txt

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt, 5304-v7.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5304) Pluggable split key policy


 [ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5304:
-

Status: Open  (was: Patch Available)

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt, 5304-v7.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5304) Pluggable split key policy


 [ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5304:
-

Status: Patch Available  (was: Open)

Getting another test run.
I also ran TestInstantSchemaChangeSplit in a loop for a bit.

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt, 5304-v7.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy

[
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199197#comment-13199197
]

Hadoop QA commented on HBASE-5304:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513024/5304-v7.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated -136 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 156 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestHFileBlock
org.apache.hadoop.hbase.client.TestAdmin

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/896//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/896//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/896//console

This message is automatically generated.

Pluggable split key policy
--

Key: HBASE-5304
URL: https://issues.apache.org/jira/browse/HBASE-5304
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.94.0

Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt, 5304-v7.txt

We need a way to specify custom policies to determine split keys.

[jira] [Updated] (HBASE-4658) Put attributes are not exposed via the ThriftServer


 [ 
https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4658:
---

Attachment: D1563.3.patch

dhruba updated the revision [jira] [HBASE-4658] Put attributes are not exposed 
via the ThriftServer.
Reviewers: tedyu, sc

  Incorporated feedback from tedyu. Thanks.

REVISION DETAIL
  https://reviews.facebook.net/D1563

AFFECTED FILES
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
  src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift


 Put attributes are not exposed via the ThriftServer
 ---

 Key: HBASE-4658
 URL: https://issues.apache.org/jira/browse/HBASE-4658
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1563.1.patch, D1563.1.patch, D1563.1.patch, 
 D1563.2.patch, D1563.2.patch, D1563.2.patch, D1563.3.patch, 
 ThriftPutAttributes1.txt


 The Put api also takes in a bunch of arbitrary attributes that an application 
 can use to associate metadata with each put operation. This is not exposed 
 via Thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4658) Put attributes are not exposed via the ThriftServer

2012-02-02 Thread dhruba borthakur (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dhruba borthakur updated HBASE-4658:


Status: Patch Available  (was: Open)

 Put attributes are not exposed via the ThriftServer
 ---

 Key: HBASE-4658
 URL: https://issues.apache.org/jira/browse/HBASE-4658
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1563.1.patch, D1563.1.patch, D1563.1.patch, 
 D1563.2.patch, D1563.2.patch, D1563.2.patch, D1563.3.patch, D1563.3.patch, 
 D1563.3.patch, ThriftPutAttributes1.txt


 The Put api also takes in a bunch of arbitrary attributes that an application 
 can use to associate metadata with each put operation. This is not exposed 
 via Thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4658) Put attributes are not exposed via the ThriftServer


 [ 
https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4658:
---

Attachment: D1563.3.patch

dhruba updated the revision [jira] [HBASE-4658] Put attributes are not exposed 
via the ThriftServer.
Reviewers: tedyu, sc

  Incorporated feedback from tedyu. Thanks.

REVISION DETAIL
  https://reviews.facebook.net/D1563

AFFECTED FILES
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
  src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift


 Put attributes are not exposed via the ThriftServer
 ---

 Key: HBASE-4658
 URL: https://issues.apache.org/jira/browse/HBASE-4658
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1563.1.patch, D1563.1.patch, D1563.1.patch, 
 D1563.2.patch, D1563.2.patch, D1563.2.patch, D1563.3.patch, D1563.3.patch, 
 D1563.3.patch, ThriftPutAttributes1.txt


 The Put api also takes in a bunch of arbitrary attributes that an application 
 can use to associate metadata with each put operation. This is not exposed 
 via Thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4658) Put attributes are not exposed via the ThriftServer


 [ 
https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4658:
---

Attachment: D1563.3.patch

dhruba updated the revision [jira] [HBASE-4658] Put attributes are not exposed 
via the ThriftServer.
Reviewers: tedyu, sc

  Incorporated feedback from tedyu. Thanks.

REVISION DETAIL
  https://reviews.facebook.net/D1563

AFFECTED FILES
  src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
  src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java
  src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift


 Put attributes are not exposed via the ThriftServer
 ---

 Key: HBASE-4658
 URL: https://issues.apache.org/jira/browse/HBASE-4658
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1563.1.patch, D1563.1.patch, D1563.1.patch, 
 D1563.2.patch, D1563.2.patch, D1563.2.patch, D1563.3.patch, D1563.3.patch, 
 D1563.3.patch, ThriftPutAttributes1.txt


 The Put api also takes in a bunch of arbitrary attributes that an application 
 can use to associate metadata with each put operation. This is not exposed 
 via Thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-5287) [89-fb] hbck can go into an infinite loop

2012-02-02 Thread Prakash Khemani (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Khemani resolved HBASE-5287.


Resolution: Fixed

Mikhail is uploading patch

 [89-fb] hbck can go into an infinite loop
 -

 Key: HBASE-5287
 URL: https://issues.apache.org/jira/browse/HBASE-5287
 Project: HBase
  Issue Type: Bug
Reporter: Prakash Khemani

 HBaseFsckRepair.prompt() should check for -1 return value from 
 System.in.read()
 Only affects 0.89 release.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5287) [89-fb] hbck can go into an infinite loop

2012-02-02 Thread Prakash Khemani (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Khemani updated HBASE-5287:
---

Summary: [89-fb] hbck can go into an infinite loop  (was: hbck can go into 
an infinite loop)

will close this issue. Mikhail will be uploading the patch separately.

 [89-fb] hbck can go into an infinite loop
 -

 Key: HBASE-5287
 URL: https://issues.apache.org/jira/browse/HBASE-5287
 Project: HBase
  Issue Type: Bug
Reporter: Prakash Khemani

 HBaseFsckRepair.prompt() should check for -1 return value from 
 System.in.read()
 Only affects 0.89 release.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4658) Put attributes are not exposed via the ThriftServer

[
https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199248#comment-13199248
]

Hadoop QA commented on HBASE-4658:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513034/D1563.3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated -140 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 157 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationPeer
org.apache.hadoop.hbase.io.hfile.TestHFileBlock
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/897//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/897//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/897//console

This message is automatically generated.

Put attributes are not exposed via the ThriftServer
---

Key: HBASE-4658
URL: https://issues.apache.org/jira/browse/HBASE-4658
Project: HBase
Issue Type: Bug
Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Attachments: D1563.1.patch, D1563.1.patch, D1563.1.patch,
D1563.2.patch, D1563.2.patch, D1563.2.patch, D1563.3.patch, D1563.3.patch,
D1563.3.patch, ThriftPutAttributes1.txt

The Put api also takes in a bunch of arbitrary attributes that an application
can use to associate metadata with each put operation. This is not exposed
via Thrift.

[jira] [Commented] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well


[ 
https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199255#comment-13199255
 ] 

Phabricator commented on HBASE-5292:


mbautin has commented on the revision [jira] [HBASE-5292] [89-fb] Prevent 
counting getSize on compacations.

  Please fix the misspelling in the title :)
  Have not really reviewed the diff yet.

REVISION DETAIL
  https://reviews.facebook.net/D1527


 getsize per-CF metric incorrectly counts compaction related reads as well 
 --

 Key: HBASE-5292
 URL: https://issues.apache.org/jira/browse/HBASE-5292
 Project: HBase
  Issue Type: Bug
Reporter: Kannan Muthukkaruppan
 Attachments: D1527.1.patch, D1527.2.patch


 The per-CF getsize metric's intent was to track bytes returned (to HBase 
 clients) per-CF. [Note: We already have metrics to track # of HFileBlock's 
 read for compaction vs. non-compaction cases -- e.g., compactionblockreadcnt 
 vs. fsblockreadcnt.]
 Currently, the getsize metric gets updated for both client initiated 
 Get/Scan operations as well for compaction related reads. The metric is 
 updated in StoreScanner.java:next() when the Scan query matcher returns an 
 INCLUDE* code via a:
  HRegion.incrNumericMetric(this.metricNameGetsize, copyKv.getLength());
 We should not do the above in case of compactions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well

[
https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199258#comment-13199258
]

Phabricator commented on HBASE-5292:

zhiqiu has commented on the revision [jira] [HBASE-5292] [89-fb] Prevent
counting getSize on compactions.

Fixed. Thx a lot!

REVISION DETAIL
https://reviews.facebook.net/D1527

getsize per-CF metric incorrectly counts compaction related reads as well
--

Key: HBASE-5292
URL: https://issues.apache.org/jira/browse/HBASE-5292
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
Attachments: D1527.1.patch, D1527.2.patch

The per-CF getsize metric's intent was to track bytes returned (to HBase
clients) per-CF. [Note: We already have metrics to track # of HFileBlock's
read for compaction vs. non-compaction cases -- e.g., compactionblockreadcnt
vs. fsblockreadcnt.]
Currently, the getsize metric gets updated for both client initiated
Get/Scan operations as well for compaction related reads. The metric is
updated in StoreScanner.java:next() when the Scan query matcher returns an
INCLUDE* code via a:
HRegion.incrNumericMetric(this.metricNameGetsize, copyKv.getLength());
We should not do the above in case of compactions.

[jira] [Commented] (HBASE-4658) Put attributes are not exposed via the ThriftServer


[ 
https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199263#comment-13199263
 ] 

Zhihong Yu commented on HBASE-4658:
---

Test failures are unrelated.

+1 on patch v3.

 Put attributes are not exposed via the ThriftServer
 ---

 Key: HBASE-4658
 URL: https://issues.apache.org/jira/browse/HBASE-4658
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: D1563.1.patch, D1563.1.patch, D1563.1.patch, 
 D1563.2.patch, D1563.2.patch, D1563.2.patch, D1563.3.patch, D1563.3.patch, 
 D1563.3.patch, ThriftPutAttributes1.txt


 The Put api also takes in a bunch of arbitrary attributes that an application 
 can use to associate metadata with each put operation. This is not exposed 
 via Thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5324) Hbck fix dryrun/plan

2012-02-02 Thread Jimmy Xiang (Created) (JIRA)

Hbck fix dryrun/plan 
-

 Key: HBASE-5324
 URL: https://issues.apache.org/jira/browse/HBASE-5324
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.94.0
Reporter: Jimmy Xiang


Hbck fix should have a dryrun option, or show the planed operations/steps at 
first, then start the actual fix after confirmed by the user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5266) Add documentation for ColumnRangeFilter


[ 
https://issues.apache.org/jira/browse/HBASE-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199271#comment-13199271
 ] 

Hudson commented on HBASE-5266:
---

Integrated in HBase-TRUNK #2652 (See 
[https://builds.apache.org/job/HBase-TRUNK/2652/])
HBASE-5266 Add documentation for ColumnRangeFilter

larsh : 
Files : 
* /hbase/trunk/src/docbkx/book.xml


 Add documentation for ColumnRangeFilter
 ---

 Key: HBASE-5266
 URL: https://issues.apache.org/jira/browse/HBASE-5266
 Project: HBase
  Issue Type: Sub-task
  Components: documentation
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.94.0

 Attachments: 5266-v2.txt, 5266-v3.txt, 5266.txt


 There are only a few lines of documentation for ColumnRangeFilter.
 Given the usefulness of this filter for efficient intra-row scanning (see 
 HBASE-5229 and HBASE-4256), we should make this filter more prominent in the 
 documentation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5324) Hbck fix dryrun/plan

2012-02-02 Thread Zhihong Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5324:
--

  Description: Hbck fix should have a dryrun option, or show the planned 
operations/steps at first, then start the actual fix after confirmed by the 
user.  (was: Hbck fix should have a dryrun option, or show the planed 
operations/steps at first, then start the actual fix after confirmed by the 
user.)
Fix Version/s: 0.94.0

 Hbck fix dryrun/plan 
 -

 Key: HBASE-5324
 URL: https://issues.apache.org/jira/browse/HBASE-5324
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.94.0
Reporter: Jimmy Xiang
 Fix For: 0.94.0


 Hbck fix should have a dryrun option, or show the planned operations/steps at 
 first, then start the actual fix after confirmed by the user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5186) Add metrics to ThriftServer


[ 
https://issues.apache.org/jira/browse/HBASE-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199272#comment-13199272
 ] 

Hudson commented on HBASE-5186:
---

Integrated in HBase-TRUNK #2652 (See 
[https://builds.apache.org/job/HBase-TRUNK/2652/])
HBASE-5186 Add metrics to ThriftServer (Scott Chen)

tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/CallQueue.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/HbaseHandlerMetricsProxy.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/ThriftMetrics.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestCallQueue.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java


 Add metrics to ThriftServer
 ---

 Key: HBASE-5186
 URL: https://issues.apache.org/jira/browse/HBASE-5186
 Project: HBase
  Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
 Fix For: 0.94.0

 Attachments: 5186-v10.txt, 5186-v11.txt, 5186-v12.txt, 5186-v9.txt, 
 HBASE-5186.D1461.1.patch, HBASE-5186.D1461.2.patch, HBASE-5186.D1461.3.patch, 
 HBASE-5186.D1461.4.patch, HBASE-5186.D1461.5.patch, HBASE-5186.D1461.6.patch, 
 HBASE-5186.D1461.7.patch, HBASE-5186.D1461.8.patch


 It will be useful to have some metrics (queue length, waiting time, 
 processing time ...) similar to Hadoop RPC server. This allows us to monitor 
 system health also provide a tool to diagnose the problem where thrift calls 
 are slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199280#comment-13199280
 ] 

Lars Hofhansl commented on HBASE-5304:
--

Ran test admin locally... passes fine. Do I get a +1 Ted? :)

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt, 5304-v7.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199284#comment-13199284
 ] 

Zhihong Yu commented on HBASE-5304:
---

Good to go.

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt, 5304-v7.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

2012-02-02 Thread Mubarak Seyed (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199301#comment-13199301
 ] 

Mubarak Seyed commented on HBASE-4528:
--

Is there any plan on backporting to 92.1 and/or 0.90.7?

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans

2012-02-02 Thread Hitesh Shah (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199304#comment-13199304
 ] 

Hitesh Shah commented on HBASE-5325:


Today, all that is exposed is the version information for hadoop and hbase. It 
might be useful to expose the additional information that is also displayed on 
the /master-status page.

 Expose basic information about the master-status through jmx beans 
 ---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Bug
Reporter: Hitesh Shah
Priority: Minor

 Similar to the Namenode and Jobtracker, it would be good if the hbase master 
 could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5325) Expose basic information about the master-status through jmx beans

2012-02-02 Thread Hitesh Shah (Created) (JIRA)

Expose basic information about the master-status through jmx beans 
---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Bug
Reporter: Hitesh Shah
Priority: Minor


Similar to the Namenode and Jobtracker, it would be good if the hbase master 
could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well

2012-02-02 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199307#comment-13199307
]

Phabricator commented on HBASE-5292:

Kannan has commented on the revision [jira] [HBASE-5292] [89-fb] Prevent
counting getSize on compactions.

Zhiqui: Looks very good. A few comments inlined...

INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:299 i think
we can avoid this hashset and contains check later.

Since this is all internal code, we can assume that if caller pass a non-null
metric name, it is good to use.
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java:200 we
need to a append a . here for compatibility with old code, and for
readability.

Otherwise, the metric names will be
cf.column_familygetsize
instead of
cf.column_family.getsize.

src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java:136
nice test!
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java:345 as
mentioned earlier, I think we can get rid of the hashset.

Also, can we pass in null (instead of ) for all the places you don't want
this metric to be tracked. And this check would simply become:

if (metric != null)

REVISION DETAIL
https://reviews.facebook.net/D1527

getsize per-CF metric incorrectly counts compaction related reads as well
--

Key: HBASE-5292
URL: https://issues.apache.org/jira/browse/HBASE-5292
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
Attachments: D1527.1.patch, D1527.2.patch

[jira] [Commented] (HBASE-5128) [uber hbck] Enable hbck to automatically repair table integrity problems as well as region consistency problems while online.


[ 
https://issues.apache.org/jira/browse/HBASE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199310#comment-13199310
 ] 

jirapos...@reviews.apache.org commented on HBASE-5128:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3435/#review4781
---



src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/3435/#comment10510

Is this wrong? Should it be  0 here and  0 below?


- Jimmy


On 2012-01-25 17:24:41, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3435/
bq.  ---
bq.  
bq.  (Updated 2012-01-25 17:24:41)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, and 
Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  I'm posting a preliminary version that I'm currently testing on real 
clusters. The tests are flakey on the 0.90 branch (so there is something async 
that I didn't synchronize properly), and there are a few more TODO's I want to 
knock out before this is ready for full review to be considered for committing. 
It's got some problems I need some advice figuring out.
bq.  
bq.  Problem 1:
bq.  
bq.  In the unit tests, I have a few cases where I fabricate new regions and 
try to force the overlapping regions to be closed. For some of these, I cannot 
delete a table after it is repaired without causing subsequent tests to fail. I 
think this is due to a few things:
bq.  
bq.  1) The disable table handler uses in-memory assignment manager state while 
delete uses in META assignment information.
bq.  2) Currently I'm using the sneaky closeRegion that purposely doesn't go 
through the master and in turn doesn't modify in-memory state – disable uses 
out of date in-memory region assignments. If I use the unassign method sends 
RIT transitions to the master, but which ends up attempting to assign it again, 
causing timing/transient states.
bq.  
bq.  What is a good way to clear the HMaster's assignment manager's assignment 
data for particular regions or to force it to re-read from META? (without 
modifying the 0.90 HBase's it is meant to repair).
bq.  
bq.  Problem 2:
bq.  
bq.  Sometimes test fail reporting HOLE_IN_REGION_CHAIN and 
SERVER_DOES_NOT_MATCH_META. This means the old and new regions are confiused 
with each other and basically something is still happening asynchronously. I 
think this is the new region is being assigned and is still transitioning. 
Sound about right? To make the unit test deterministic, should hbck wait for 
these to settle or should just the unit test wait?
bq.  
bq.  
bq.  This addresses bug HBASE-5128.
bq.  https://issues.apache.org/jira/browse/HBASE-5128
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java c56b3a6 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
9520b95 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f7ad064 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 6d3401d 
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java a3d8b8b 
bq.src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
29e8bb2 
bq.
src/main/java/org/apache/hadoop/hbase/util/hbck/TableIntegrityErrorHandler.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 7138d63 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a640d57 
bq.src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckComparator.java 
2c4a79e 
bq.src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java 
dbb97f8 
bq.
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java 
11a1151 
bq.  
bq.  Diff: https://reviews.apache.org/r/3435/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All unit tests pass sometimes.  Some fail sometimes (generally the cases 
that fabricate new regions).  
bq.  
bq.  Not ready for commit.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



 [uber hbck] Enable hbck to automatically repair table integrity problems as 
 well as region consistency problems while online.
 -

 Key: HBASE-5128
 URL: https://issues.apache.org/jira/browse/HBASE-5128
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.90.5, 0.92.0
Reporter: Jonathan Hsieh
Assignee:

[jira] [Created] (HBASE-5326) splitlogmanager zk async handlers after shutdown

2012-02-02 Thread Prakash Khemani (Created) (JIRA)

splitlogmanager zk async handlers after shutdown


 Key: HBASE-5326
 URL: https://issues.apache.org/jira/browse/HBASE-5326
 Project: HBase
  Issue Type: Improvement
Reporter: Prakash Khemani


The zk async handlers in SpltLogManager should ignore all callbacks after 
SplitLogManager has shutdown. Will make the test logs less noisy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-5326) splitlogmanager zk async handlers after shutdown

2012-02-02 Thread Prakash Khemani (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Khemani reassigned HBASE-5326:
--

Assignee: Prakash Khemani

 splitlogmanager zk async handlers after shutdown
 

 Key: HBASE-5326
 URL: https://issues.apache.org/jira/browse/HBASE-5326
 Project: HBase
  Issue Type: Improvement
Reporter: Prakash Khemani
Assignee: Prakash Khemani

 The zk async handlers in SpltLogManager should ignore all callbacks after 
 SplitLogManager has shutdown. Will make the test logs less noisy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

2012-02-02 Thread Jonathan Gray (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199312#comment-13199312
 ] 

Jonathan Gray commented on HBASE-4528:
--

@Mubarek, since it's a performance optimization and new feature, it's not going 
to be committed into the 90/92 branches.  That being said, this patch could be 
backported if someone wanted to use it on a 92 branch (90 might be 
significantly more difficult, not sure).

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-5308) Retry of distributed log splitting will fail on ./logs/rs-splitting directories

2012-02-02 Thread Prakash Khemani (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Khemani resolved HBASE-5308.


Resolution: Fixed

Mikhail will upload the internal patch to the 89-fb branch

 Retry of distributed log splitting will fail on ./logs/rs-splitting 
 directories
 ---

 Key: HBASE-5308
 URL: https://issues.apache.org/jira/browse/HBASE-5308
 Project: HBase
  Issue Type: Bug
 Environment: Only exists in 89 branch
 Master.splitLog() doesn't handle the case where the rs log file has been 
 renamed
Reporter: Prakash Khemani



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3792) TableInputFormat leaks ZK connections

2012-02-02 Thread Terry Siu (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199325#comment-13199325
]

Terry Siu commented on HBASE-3792:
--

No worries, Bryan. I managed to patch it myself a while back using the
tableinput.patch as a guideline. Thanks again!

TableInputFormat leaks ZK connections
-

Key: HBASE-3792
URL: https://issues.apache.org/jira/browse/HBASE-3792
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.90.1
Environment: Java 1.6.0_24, Mac OS X 10.6.7
Reporter: Bryan Keller
Attachments: patch0.90.4, tableinput.patch

The TableInputFormat creates an HTable using a new Configuration object, and
it never cleans it up. When running a Mapper, the TableInputFormat is
instantiated and the ZK connection is created. While this connection is not
explicitly cleaned up, the Mapper process eventually exits and thus the
connection is closed. Ideally the TableRecordReader would close the
connection in its close() method rather than relying on the process to die
for connection cleanup. This is fairly easy to implement by overriding
TableRecordReader, and also overriding TableInputFormat to specify the new
record reader.
The leak occurs when the JobClient is initializing and needs to retrieves the
splits. To get the splits, it instantiates a TableInputFormat. Doing so
creates a ZK connection that is never cleaned up. Unlike the mapper, however,
my job client process does not die. Thus the ZK connections accumulate.
I was able to fix the problem by writing my own TableInputFormat that does
not initialize the HTable in the getConf() method and does not have an HTable
member variable. Rather, it has a variable for the table name. The HTable is
instantiated where needed and then cleaned up. For example, in the
getSplits() method, I create the HTable, then close the connection once the
splits are retrieved. I also create the HTable when creating the record
reader, and I have a record reader that closes the connection when done.

[jira] [Updated] (HBASE-5325) Expose basic information about the master-status through jmx beans

2012-02-02 Thread Hitesh Shah (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah updated HBASE-5325:
---

Attachment: HBASE-5325.wip.patch

Attached is a preliminary patch which adds a new hbasemasterinfo bean. Please 
let me know your comments/thoughts on whether this is the right 
approach/direction. 

If it looks good, I can add something similar for the region server ( and 
tests, etc ). 

Using a hacked patch to display the /jmx output, this is what the information 
would look like: 

{quote}
{
name : Hadoop:service=HBase,name=HBaseMasterInfo,
modelerType : 
org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster,
ClusterId : d24914d7-75d3-4dcc-9e6f-0d7770833993,
ZookeeperQuorumInfo : localhost:2181,
CoprocessorsInfo : [],
MasterStartTime : 1328223257723,
MasterActiveTime : 1328223257725,
RegionServers : 
{\10.10.10.128:57366\:{\requestsPerSecond\:0,\startcode\:1328223257711,\maxHeapMB\:987,\numberOfOnlineRegions\:2,\usedHeapMB\:34}},
DeadRegionServers : [],
RegionsInTransition : {}
  }
{quote}



 Expose basic information about the master-status through jmx beans 
 ---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Bug
Reporter: Hitesh Shah
Priority: Minor
 Attachments: HBASE-5325.wip.patch


 Similar to the Namenode and Jobtracker, it would be good if the hbase master 
 could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5304) Pluggable split key policy


 [ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5304:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk.
Thanks for the reviews Jesse and Ted.

 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt, 5304-v7.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5318) Support Eclipse Indigo

[
https://issues.apache.org/jira/browse/HBASE-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199334#comment-13199334
]

Lars Hofhansl commented on HBASE-5318:
--

Looks good to me, will commit to trunk in the next 1/2 hour if nobody objects.

Support Eclipse Indigo
---

[jira] [Commented] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well

[
https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199345#comment-13199345
]

Phabricator commented on HBASE-5292:

zhiqiu has commented on the revision [jira] [HBASE-5292] [89-fb] Prevent
counting getSize on compactions.

INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:299 Sure.
Will remove this. :)
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java:200 I
checked the implementation of generateSchemaMetricsPrefix() and found that a
. is already appended to the cf name.

schemaMetricPrefix +=
cfName.equals(TOTAL_KEY) ? : CF_PREFIX + cfName + .;

Actually I removed it because in the test log, I found the name is like
tbl.SizeMetricTest.cf.cf1..getsize, w/ one extra .
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java:345 Yes,
it will be much simpler by changing it to null. Will do this right now. :D

REVISION DETAIL
https://reviews.facebook.net/D1527

getsize per-CF metric incorrectly counts compaction related reads as well
--

Key: HBASE-5292
URL: https://issues.apache.org/jira/browse/HBASE-5292
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
Attachments: D1527.1.patch, D1527.2.patch

[jira] [Commented] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well

[
https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199347#comment-13199347
]

Phabricator commented on HBASE-5292:

mbautin has commented on the revision [jira] [HBASE-5292] [89-fb] Prevent
counting getSize on compactions.

@zhiqiu: yes, please make sure we don't end up with two dots in metric names.
We have had one bug of this kind in production already. Thanks!

REVISION DETAIL
https://reviews.facebook.net/D1527

getsize per-CF metric incorrectly counts compaction related reads as well
--

Key: HBASE-5292
URL: https://issues.apache.org/jira/browse/HBASE-5292
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
Attachments: D1527.1.patch, D1527.2.patch

[jira] [Commented] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well

[
https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199350#comment-13199350
]

Phabricator commented on HBASE-5292:

Kannan has commented on the revision [jira] [HBASE-5292] [89-fb] Prevent
counting getSize on compactions.

Ok, the ..getsize issue was something Liyin fixed recently. Perhaps you are
not rebased on that.

REVISION DETAIL
https://reviews.facebook.net/D1527

getsize per-CF metric incorrectly counts compaction related reads as well
--

Key: HBASE-5292
URL: https://issues.apache.org/jira/browse/HBASE-5292
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
Attachments: D1527.1.patch, D1527.2.patch

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199351#comment-13199351
 ] 

Lars Hofhansl commented on HBASE-4528:
--

-1 on backporting. 0.94 will be branched in less than a month.
There are many, many more performance enhancement in 0.94, we cannot backport 
them all and neither should we.


 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5327) Print a message when an invalid hbase.rootdir is passed

2012-02-02 Thread Jean-Daniel Cryans (Created) (JIRA)

Print a message when an invalid hbase.rootdir is passed
---

 Key: HBASE-5327
 URL: https://issues.apache.org/jira/browse/HBASE-5327
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.5
Reporter: Jean-Daniel Cryans
 Fix For: 0.94.0, 0.90.7, 0.92.1


As seen on the mailing list: 
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/24124

If hbase.rootdir doesn't specify a folder on hdfs we crash while opening a path 
to .oldlogs:

{noformat}
2012-02-02 23:07:26,292 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled 
exception. Starting shutdown.
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path 
in absolute URI: hdfs://sv4r11s38:9100.oldlogs
at org.apache.hadoop.fs.Path.initialize(Path.java:148)
at org.apache.hadoop.fs.Path.init(Path.java:71)
at org.apache.hadoop.fs.Path.init(Path.java:50)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
at 
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:448)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:326)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: 
hdfs://sv4r11s38:9100.oldlogs
at java.net.URI.checkPath(URI.java:1787)
at java.net.URI.init(URI.java:735)
at org.apache.hadoop.fs.Path.initialize(Path.java:145)
... 6 more
{noformat}

It could also crash anywhere else, this just happens to be the first place we 
use hbase.rootdir. We need to verify that it's an actual folder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans


[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199376#comment-13199376
 ] 

Zhihong Yu commented on HBASE-5325:
---

If you can upload patch to https://reviews.apache.org, that would help reviewer 
provide better review comments.

{code}
+if (mxBean != null) {
+  MBeans.unregister(mxBean);
+}
{code}
Please assign null to mxBean after unregistration.
{code}
+mxBean = MBeans.register(HBase, HBaseMasterInfo, this);
{code}
For the second parameter, MasterInfo should be good enough.

There is no need to include year in HBaseMasterMXBean.java header.
There are extra whitespaces in this file.

 Expose basic information about the master-status through jmx beans 
 ---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Bug
Reporter: Hitesh Shah
Priority: Minor
 Attachments: HBASE-5325.wip.patch


 Similar to the Namenode and Jobtracker, it would be good if the hbase master 
 could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-5318) Support Eclipse Indigo

2012-02-02 Thread Lars Hofhansl (Resolved) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lars Hofhansl resolved HBASE-5318.
--

Resolution: Fixed

Verified on my Ecplipse...

Committed to trunk. Thanks for the patch Jesse.

Support Eclipse Indigo
---

[jira] [Updated] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well


 [ 
https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5292:
---

Attachment: D1527.3.patch

zhiqiu updated the revision [jira] [HBASE-5292] [89-fb] Prevent counting 
getSize on compactions.
Reviewers: Kannan, Liyin, JIRA

  Address Kannan's comments:
  * Removed the metric name Hashset
  * Re-based on Liyin's fix of ..getsize

REVISION DETAIL
  https://reviews.facebook.net/D1527

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  src/main/java/org/apache/hadoop/hbase/regionserver/InternalScanner.java
  src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java


 getsize per-CF metric incorrectly counts compaction related reads as well 
 --

 Key: HBASE-5292
 URL: https://issues.apache.org/jira/browse/HBASE-5292
 Project: HBase
  Issue Type: Bug
Reporter: Kannan Muthukkaruppan
 Attachments: D1527.1.patch, D1527.2.patch, D1527.3.patch


 The per-CF getsize metric's intent was to track bytes returned (to HBase 
 clients) per-CF. [Note: We already have metrics to track # of HFileBlock's 
 read for compaction vs. non-compaction cases -- e.g., compactionblockreadcnt 
 vs. fsblockreadcnt.]
 Currently, the getsize metric gets updated for both client initiated 
 Get/Scan operations as well for compaction related reads. The metric is 
 updated in StoreScanner.java:next() when the Scan query matcher returns an 
 INCLUDE* code via a:
  HRegion.incrNumericMetric(this.metricNameGetsize, copyKv.getLength());
 We should not do the above in case of compactions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4336) Convert source tree into maven modules


[ 
https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199394#comment-13199394
 ] 

Jesse Yates commented on HBASE-4336:


So working through some of the packages, there are a lot of dependencies 
between files for things like constants or static methods. Moving the whole 
file into the shared directory for just these couple of things seems excessive 
(why should the client need to know about the hfile?). 

I'm proposing to follow two general ideas to resolve this:
1) Abstracting out the static methods into a Util class of the same name (so it 
would be something like HLogUtils for static method in HLog).
2) Moving super general constants to HConstants (like HFile.DEFAULT_BLOCK_SIZE, 
which is scattered liberally throughout) and then more specific constants to 
subclasses within HConstants, meaning you might get something like 
HConstants.HLog.SPLIT_SKIP_ERRORS_DEFAULT. 

My other idea for (2) was to have a constants package that just has the 
constants for each class. This second means smaller files, but less 
easy/immediate/natural access to the constants, but gives you nice separation 
between the constants.

Thoughts?

If no one disagrees, I'll precede with the described methodology in (1) and (2).

 Convert source tree into maven modules
 --

 Key: HBASE-4336
 URL: https://issues.apache.org/jira/browse/HBASE-4336
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.94.0


 When we originally converted the build to maven we had a single core module 
 defined, but later reverted this to a module-less build for the sake of 
 simplicity.
 It now looks like it's time to re-address this, as we have an actual need for 
 modules to:
 * provide a trimmed down client library that applications can make use of
 * more cleanly support building against different versions of Hadoop, in 
 place of some of the reflection machinations currently required
 * incorporate the secure RPC engine that depends on some secure Hadoop classes
 I propose we start simply by refactoring into two initial modules:
 * core - common classes and utilities, and client-side code and interfaces
 * server - master and region server implementations and supporting code
 This would also lay the groundwork for incorporating the HBase security 
 features that have been developed.  Once the module structure is in place, 
 security-related features could then be incorporated into a third module -- 
 security -- after normal review and approval.  The security module could 
 then depend on secure Hadoop, without modifying the dependencies of the rest 
 of the HBase code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4336) Convert source tree into maven modules


[ 
https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199402#comment-13199402
 ] 

Zhihong Yu commented on HBASE-4336:
---

#1 is a good idea.
For #2, please keep us posted on what constants to move to HConstants. Earlier 
there was discussion on the scope of certain constants.

 Convert source tree into maven modules
 --

 Key: HBASE-4336
 URL: https://issues.apache.org/jira/browse/HBASE-4336
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.94.0


 When we originally converted the build to maven we had a single core module 
 defined, but later reverted this to a module-less build for the sake of 
 simplicity.
 It now looks like it's time to re-address this, as we have an actual need for 
 modules to:
 * provide a trimmed down client library that applications can make use of
 * more cleanly support building against different versions of Hadoop, in 
 place of some of the reflection machinations currently required
 * incorporate the secure RPC engine that depends on some secure Hadoop classes
 I propose we start simply by refactoring into two initial modules:
 * core - common classes and utilities, and client-side code and interfaces
 * server - master and region server implementations and supporting code
 This would also lay the groundwork for incorporating the HBase security 
 features that have been developed.  Once the module structure is in place, 
 security-related features could then be incorporated into a third module -- 
 security -- after normal review and approval.  The security module could 
 then depend on secure Hadoop, without modifying the dependencies of the rest 
 of the HBase code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4336) Convert source tree into maven modules


[ 
https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199432#comment-13199432
 ] 

Zhihong Yu commented on HBASE-4336:
---

@Jesse:
I think you should refresh your workspace. The following query doesn't return 
anything in TRUNK:
{code}
zhihyu$ find src/main -name '*.java' -exec grep -i DEFAULT_BLOCK_S {} \; -print
{code}

 Convert source tree into maven modules
 --

 Key: HBASE-4336
 URL: https://issues.apache.org/jira/browse/HBASE-4336
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.94.0


 When we originally converted the build to maven we had a single core module 
 defined, but later reverted this to a module-less build for the sake of 
 simplicity.
 It now looks like it's time to re-address this, as we have an actual need for 
 modules to:
 * provide a trimmed down client library that applications can make use of
 * more cleanly support building against different versions of Hadoop, in 
 place of some of the reflection machinations currently required
 * incorporate the secure RPC engine that depends on some secure Hadoop classes
 I propose we start simply by refactoring into two initial modules:
 * core - common classes and utilities, and client-side code and interfaces
 * server - master and region server implementations and supporting code
 This would also lay the groundwork for incorporating the HBase security 
 features that have been developed.  Once the module structure is in place, 
 security-related features could then be incorporated into a third module -- 
 security -- after normal review and approval.  The security module could 
 then depend on secure Hadoop, without modifying the dependencies of the rest 
 of the HBase code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans

2012-02-02 Thread Hitesh Shah (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199439#comment-13199439
 ] 

Hitesh Shah commented on HBASE-5325:


Thanks for the quick look, @Zhihong. If you don't have any major concerns, I 
will go ahead and provide a full patch for review with tests for both the 
master and region server implementations. If you do see something which strikes 
you, I would like to get a better understanding about your concerns before I 
start making changes for the region server. Please let me know accordingly. 

 Expose basic information about the master-status through jmx beans 
 ---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Bug
Reporter: Hitesh Shah
Priority: Minor
 Attachments: HBASE-5325.wip.patch


 Similar to the Namenode and Jobtracker, it would be good if the hbase master 
 could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5229) Explore building blocks for multi-row local transactions.

2012-02-02 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lars Hofhansl updated HBASE-5229:
-

Attachment: 5229-multiRow.txt

Patch that works for me. Looks bigger than it is. Just refactors some code to
make it usable for MultiRow ops.

Advanced use only is prominently mentioned in the Javadoc, and it also
advises to either disable splitting or use a custom RegionSplitPolicy.

Explore building blocks for multi-row local transactions.
---

Key: HBASE-5229
URL: https://issues.apache.org/jira/browse/HBASE-5229
Project: HBase
Issue Type: New Feature
Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.94.0

Attachments: 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt,
5229.txt

HBase should provide basic building blocks for multi-row local transactions.
Local means that we do this by co-locating the data. Global (cross region)
transactions are not discussed here.
After a bit of discussion two solutions have emerged:
1. Keep the row-key for determining grouping and location and allow efficient
intra-row scanning. A client application would then model tables as
HBase-rows.
2. Define a prefix-length in HTableDescriptor that defines a grouping of
rows. Regions will then never be split inside a grouping prefix.
#1 is true to the current storage paradigm of HBase.
#2 is true to the current client side API.
I will explore these two with sample patches here.

Was:
As discussed (at length) on the dev mailing list with the HBASE-3584 and
HBASE-5203 committed, supporting atomic cross row transactions within a
region becomes simple.
I am aware of the hesitation about the usefulness of this feature, but we
have to start somewhere.
Let's use this jira for discussion, I'll attach a patch (with tests)
momentarily to make this concrete.

[jira] [Commented] (HBASE-5229) Explore building blocks for multi-row local transactions.

[
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199459#comment-13199459
]

jirapos...@reviews.apache.org commented on HBASE-5229:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/
---

Review request for hbase.

Summary
---

This builds on HBASE-3584, HBASE-5203, and HBASE-5304.

Multiple Rows can be locked and applied atomically as long as the application
ensures that all rows reside in the same Region (by presplitting or a custom
RegionSplitPolicy).
At SFDC we can use this to colocate subsets of a tenant's data and allow atomic
operations over these subsets.

Obviously this is an advanced features and this prominently called out in the
Javadoc.

This addresses bug HBASE-5229.
https://issues.apache.org/jira/browse/HBASE-5229

Diffs
-

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
PRE-CREATION

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
1239953

http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
1239953

Diff: https://reviews.apache.org/r/3748/diff

Testing
---

Tests added to TestFromClientSide and TestAtomicOperation

Thanks,

Lars

Explore building blocks for multi-row local transactions.
---

Attachments: 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt,
5229.txt

[jira] [Commented] (HBASE-4336) Convert source tree into maven modules


[ 
https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199462#comment-13199462
 ] 

Jesse Yates commented on HBASE-4336:


@Zhihong - im worried that a lot of the work will go to waste in doing the 
refresh. I'm working more on seeing it if is possible at an older point (or 
even close) and then seeing how that translates to trunk. I think I'm a few 
weeks behind trunk, which shouldn't be _too_ bad

 Convert source tree into maven modules
 --

 Key: HBASE-4336
 URL: https://issues.apache.org/jira/browse/HBASE-4336
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.94.0


 When we originally converted the build to maven we had a single core module 
 defined, but later reverted this to a module-less build for the sake of 
 simplicity.
 It now looks like it's time to re-address this, as we have an actual need for 
 modules to:
 * provide a trimmed down client library that applications can make use of
 * more cleanly support building against different versions of Hadoop, in 
 place of some of the reflection machinations currently required
 * incorporate the secure RPC engine that depends on some secure Hadoop classes
 I propose we start simply by refactoring into two initial modules:
 * core - common classes and utilities, and client-side code and interfaces
 * server - master and region server implementations and supporting code
 This would also lay the groundwork for incorporating the HBase security 
 features that have been developed.  Once the module structure is in place, 
 security-related features could then be incorporated into a third module -- 
 security -- after normal review and approval.  The security module could 
 then depend on secure Hadoop, without modifying the dependencies of the rest 
 of the HBase code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans


[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199477#comment-13199477
 ] 

Zhihong Yu commented on HBASE-5325:
---

I don't have major concern.
I think we should mention JSON format for return values of several methods in 
HBaseMasterMXBean.

Of course, other people's opinions are welcome.

 Expose basic information about the master-status through jmx beans 
 ---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Bug
Reporter: Hitesh Shah
Priority: Minor
 Attachments: HBASE-5325.wip.patch


 Similar to the Namenode and Jobtracker, it would be good if the hbase master 
 could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5292) getsize per-CF metric incorrectly counts compaction related reads as well

2012-02-02 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199484#comment-13199484
 ] 

Phabricator commented on HBASE-5292:


Kannan has accepted the revision [jira] [HBASE-5292] [89-fb] Prevent counting 
getSize on compactions.

  nice work Zhiqui!

REVISION DETAIL
  https://reviews.facebook.net/D1527


 getsize per-CF metric incorrectly counts compaction related reads as well 
 --

 Key: HBASE-5292
 URL: https://issues.apache.org/jira/browse/HBASE-5292
 Project: HBase
  Issue Type: Bug
Reporter: Kannan Muthukkaruppan
 Attachments: D1527.1.patch, D1527.2.patch, D1527.3.patch


 The per-CF getsize metric's intent was to track bytes returned (to HBase 
 clients) per-CF. [Note: We already have metrics to track # of HFileBlock's 
 read for compaction vs. non-compaction cases -- e.g., compactionblockreadcnt 
 vs. fsblockreadcnt.]
 Currently, the getsize metric gets updated for both client initiated 
 Get/Scan operations as well for compaction related reads. The metric is 
 updated in StoreScanner.java:next() when the Scan query matcher returns an 
 INCLUDE* code via a:
  HRegion.incrNumericMetric(this.metricNameGetsize, copyKv.getLength());
 We should not do the above in case of compactions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3792) TableInputFormat leaks ZK connections

2012-02-02 Thread Bryan Keller (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199488#comment-13199488
]

Bryan Keller commented on HBASE-3792:
-

The latest Cloudera code introduced the new reference counting connection
management. There is a reference counter leak it appears in the HTable
constructor, thus you'll see connection leaks again and my patch doesn't fix
it. As a hack for now I force the connection to close by using reflection,
setting the ref counter to 1, and calling close() on the connection. I do this
after calling table.close() in TableInputFormat, TableRecordReader, and
TableOutputFormat. I think I should log another bug, as the leak is not in the
map reduce classes.

TableInputFormat leaks ZK connections
-

[jira] [Commented] (HBASE-5229) Explore building blocks for multi-row local transactions.


[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199514#comment-13199514
 ] 

jirapos...@reviews.apache.org commented on HBASE-5229:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4788
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
https://reviews.apache.org/r/3748/#comment10525

Should be ' to mutateRows'



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
https://reviews.apache.org/r/3748/#comment10526

Should read ' the mutations'



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
https://reviews.apache.org/r/3748/#comment10529

Can the two add() methods be combined into one which accepts Mutation ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
https://reviews.apache.org/r/3748/#comment10527

Is this method thread-safe ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
https://reviews.apache.org/r/3748/#comment10528

This comment can be removed, right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
https://reviews.apache.org/r/3748/#comment10531

From its name, RowMutation seems to refer to single row. It is a little 
confusing RowMutation extends MultiRowMutation.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
https://reviews.apache.org/r/3748/#comment10530

version would be read / written twice, right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/3748/#comment10532

Should be 'within the region', right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/3748/#comment10533

rowsToLock.size() could be smaller than mutations.size(), right ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
https://reviews.apache.org/r/3748/#comment10535

Can we refer regionName from rm (the MultiRowMutation) ?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
https://reviews.apache.org/r/3748/#comment10534

Should be mutateRows



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
https://reviews.apache.org/r/3748/#comment10536

Should read atomic mutation


- Ted


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.

[jira] [Commented] (HBASE-5229) Explore building blocks for multi-row local transactions.

2012-02-02 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199518#comment-13199518
 ] 

jirapos...@reviews.apache.org commented on HBASE-5229:
--



bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.  

The for the thourough review as usual.
I'll have a new patch tomorrow.


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 60
bq.   https://reviews.apache.org/r/3748/diff/1/?file=72039#file72039line60
bq.  
bq.   Can the two add() methods be combined into one which accepts 
Mutation ?

That is because there are other mutations that I do not support 
(Increment/Append).


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 65
bq.   https://reviews.apache.org/r/3748/diff/1/?file=72039#file72039line65
bq.  
bq.   This comment can be removed, right ?

Yes


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java,
 line 39
bq.   https://reviews.apache.org/r/3748/diff/1/?file=72040#file72040line39
bq.  
bq.   From its name, RowMutation seems to refer to single row. It is a 
little confusing RowMutation extends MultiRowMutation.

Yeah. Maybe I'll have a common super class instead.


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java,
 line 96
bq.   https://reviews.apache.org/r/3748/diff/1/?file=72040#file72040line96
bq.  
bq.   version would be read / written twice, right ?

Yes. Need to fix that.


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 4211
bq.   https://reviews.apache.org/r/3748/diff/1/?file=72044#file72044line4211
bq.  
bq.   rowsToLock.size() could be smaller than mutations.size(), right ?

Oh yes... Good point.


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java,
 line 3160
bq.   https://reviews.apache.org/r/3748/diff/1/?file=72045#file72045line3160
bq.  
bq.   Can we refer regionName from rm (the MultiRowMutation) ?

Yes, because all rows on the MultiRowMutation need to be in the same region. 
HTable just uses the first Mutation to find the region to the used.


bq.  On 2012-02-03 04:45:54, Ted Yu wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java,
 line 64
bq.   https://reviews.apache.org/r/3748/diff/1/?file=72039#file72039line64
bq.  
bq.   Is this method thread-safe ?

Should be. Only uses local state or protected data structures (like put, get, 
etc)


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4788
---


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java

[jira] [Commented] (HBASE-5229) Explore building blocks for multi-row local transactions.

2012-02-02 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199520#comment-13199520
 ] 

jirapos...@reviews.apache.org commented on HBASE-5229:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4789
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
https://reviews.apache.org/r/3748/#comment10537

Should we check they are for different rows here?



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
https://reviews.apache.org/r/3748/#comment10543

This is called by default.  No need to call it explicitly.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
https://reviews.apache.org/r/3748/#comment10545

This is called by default.  No need to call it explicitly.


- Jimmy


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.



 Explore building blocks for multi-row local transactions.
 ---

 Key: HBASE-5229
 URL: https://issues.apache.org/jira/browse/HBASE-5229
 Project: HBase
  Issue Type: New Feature
  Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 
 5229.txt


 HBase should provide basic building blocks for multi-row local transactions. 
 Local means that we do this by co-locating the data. Global (cross region) 
 transactions are not discussed here.
 After a bit of discussion two solutions have emerged:
 1. Keep the row-key for determining grouping and location and allow efficient 
 intra-row scanning. A client application would then model tables as 
 HBase-rows.
 2. Define a prefix-length in HTableDescriptor that defines a grouping of

[jira] [Commented] (HBASE-5229) Explore building blocks for multi-row local transactions.

2012-02-02 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199523#comment-13199523
 ] 

jirapos...@reviews.apache.org commented on HBASE-5229:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3748/#review4791
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
https://reviews.apache.org/r/3748/#comment10547

My point was we shouldn't throw exception in this case.
MultiRowMutation should be delivered to the correct region.

I think we agree on the above.


- Ted


On 2012-02-03 01:29:58, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3748/
bq.  ---
bq.  
bq.  (Updated 2012-02-03 01:29:58)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This builds on HBASE-3584, HBASE-5203, and HBASE-5304.
bq.  
bq.  Multiple Rows can be locked and applied atomically as long as the 
application ensures that all rows reside in the same Region (by presplitting or 
a custom RegionSplitPolicy).
bq.  At SFDC we can use this to colocate subsets of a tenant's data and allow 
atomic operations over these subsets.
bq.  
bq.  Obviously this is an advanced features and this prominently called out in 
the Javadoc.
bq.  
bq.  
bq.  This addresses bug HBASE-5229.
bq.  https://issues.apache.org/jira/browse/HBASE-5229
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
 1239953 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
 1239953 
bq.  
bq.  Diff: https://reviews.apache.org/r/3748/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tests added to TestFromClientSide and TestAtomicOperation
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.



 Explore building blocks for multi-row local transactions.
 ---

 Key: HBASE-5229
 URL: https://issues.apache.org/jira/browse/HBASE-5229
 Project: HBase
  Issue Type: New Feature
  Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 
 5229.txt


 HBase should provide basic building blocks for multi-row local transactions. 
 Local means that we do this by co-locating the data. Global (cross region) 
 transactions are not discussed here.
 After a bit of discussion two solutions have emerged:
 1. Keep the row-key for determining grouping and location and allow efficient 
 intra-row scanning. A client application would then model tables as 
 HBase-rows.
 2. Define a prefix-length in HTableDescriptor that defines a grouping of 
 rows. Regions will then never be split inside a grouping prefix.
 #1 is true to the current storage paradigm of HBase.
 #2 is true to the current client side API.
 I will explore these two with sample patches here.
 
 Was:
 As discussed (at length) on the dev mailing list with the HBASE-3584 and 
 HBASE-5203

[jira] [Commented] (HBASE-5324) Hbck fix dryrun/plan

2012-02-02 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199522#comment-13199522
 ] 

Jonathan Hsieh commented on HBASE-5324:
---

+1. Great idea.  hbck in HBASE-5128 can be dangerous and this could let users 
see if there are any unexpected fixes.

 Hbck fix dryrun/plan 
 -

 Key: HBASE-5324
 URL: https://issues.apache.org/jira/browse/HBASE-5324
 Project: HBase
  Issue Type: New Feature
  Components: hbck
Affects Versions: 0.94.0
Reporter: Jimmy Xiang
 Fix For: 0.94.0


 Hbck fix should have a dryrun option, or show the planned operations/steps at 
 first, then start the actual fix after confirmed by the user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3792) TableInputFormat leaks ZK connections

2012-02-02 Thread stack (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199528#comment-13199528
]

stack commented on HBASE-3792:
--

@Bryan Yes please. Thats crazy acrobatics you are at just to close a
connection. Thanks.

TableInputFormat leaks ZK connections
-

[jira] [Commented] (HBASE-5319) TRUNK broke since hbase-4218 went in? TestHFileBlock OOMEs


[ 
https://issues.apache.org/jira/browse/HBASE-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199531#comment-13199531
 ] 

Zhihong Yu commented on HBASE-5319:
---

Maybe the test failure implies some bug.

We should include the value of cacheOnWrite in the following log:
{code}
  LOG.info(Compression algorithm:  + algo + , pread= + pread);
{code}
From 
https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/:
{code}
2012-02-02 21:52:54,017 INFO  [pool-1-thread-1] hfile.TestHFileBlock(472): 
Compression algorithm: GZ, pread=true
2012-02-02 21:52:55,290 INFO  [pool-1-thread-1] hbase.ResourceChecker(145): 
after io.hfile.TestHFileBlock#testPreviousOffset[1]: 57 threads (was 57), 77 
file descriptors (was 75). 0 connections,  -file handle leak?- 
{code}
Looks like the OOME happened for 'Compression algorithm: GZ, pread=true, 
cacheOnWrite=true'

For the above case, we have 2 options:
1. ask for access to 
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/898f8b80-7820-4c5a-bc9b-f0fe0891bec2/prev_offset
 on build machine
2. before OOME, try to dump the contents of prev_offset

We should have more clue once we get the file that causes OOME when deflating.

 TRUNK broke since hbase-4218 went in?  TestHFileBlock OOMEs
 ---

 Key: HBASE-5319
 URL: https://issues.apache.org/jira/browse/HBASE-5319
 Project: HBase
  Issue Type: Bug
Reporter: stack

 Check it out...https://builds.apache.org/job/HBase-TRUNK/  Mikhail, you might 
 know whats up.  Else, will have a looksee...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans

2012-02-02 Thread stack (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199534#comment-13199534
 ] 

stack commented on HBASE-5325:
--

bq. Similar to the Namenode and Jobtracker, it would be good if the hbase 
master could expose some information through mbeans.

Can we go further (smile)?

Would be cool if when a regionserver registered, master started listening (or 
asking) regionservers for their metrics via jmx.  Master could use the 
collected info when it displays its UI (Perhaps it would make sense to do a 
cluster federated mbean that has the sum of all regionserver jmx metrics and 
use this displaying master ui (?) plus a master mbean with the master's 
vitals).  If we got this working, using jmx to query vitals, perhaps we could 
undo the rpc that the regionserver does to the master every second or so to 
tell it about its current load since HServerLoad is essentially duplicating 
metrics (Static or near-static properties that HServerLoad reports such as 
loaded coprocessors could be hoisted up as data in the regionservers ephemeral 
znode as protobufs/json data -- we could add to whats in HServerLoad reporting 
system vitals too like RAM, CPUs, Disks.  Master could do the same hoisting 
vitals up into its zk znode... and/or emit in an mbean).


 Expose basic information about the master-status through jmx beans 
 ---

 Key: HBASE-5325
 URL: https://issues.apache.org/jira/browse/HBASE-5325
 Project: HBase
  Issue Type: Bug
Reporter: Hitesh Shah
Priority: Minor
 Attachments: HBASE-5325.wip.patch


 Similar to the Namenode and Jobtracker, it would be good if the hbase master 
 could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5304) Pluggable split key policy


[ 
https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13199537#comment-13199537
 ] 

Hudson commented on HBASE-5304:
---

Integrated in HBase-TRUNK-security #96 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/96/])
HBASE-5304 Pluggable split key policy

larsh : 
Files : 
* /hbase/trunk/src/docbkx/book.xml
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionSplitPolicy.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/PrefixSplitKeyPolicy.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java


 Pluggable split key policy
 --

 Key: HBASE-5304
 URL: https://issues.apache.org/jira/browse/HBASE-5304
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.0

 Attachments: 5304-v4.txt, 5304-v5.txt, 5304-v6.txt, 5304-v7.txt


 We need a way to specify custom policies to determine split keys.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5318) Support Eclipse Indigo