[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5869:
-

Status: Open  (was: Patch Available)

> Move SplitLogManager splitlog taskstate and AssignmentManager 
> RegionTransitionData znode datas to pb 
> -
>
> Key: HBASE-5869
> URL: https://issues.apache.org/jira/browse/HBASE-5869
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, 
> secondcut.txt, v10.txt, v4.txt, v5.txt, v6.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5869:
-

Attachment: v10.txt

File path to recovered edits prob. fixed

> Move SplitLogManager splitlog taskstate and AssignmentManager 
> RegionTransitionData znode datas to pb 
> -
>
> Key: HBASE-5869
> URL: https://issues.apache.org/jira/browse/HBASE-5869
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, 
> secondcut.txt, v10.txt, v4.txt, v5.txt, v6.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5869:
-

Status: Patch Available  (was: Open)

> Move SplitLogManager splitlog taskstate and AssignmentManager 
> RegionTransitionData znode datas to pb 
> -
>
> Key: HBASE-5869
> URL: https://issues.apache.org/jira/browse/HBASE-5869
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, 
> secondcut.txt, v10.txt, v4.txt, v5.txt, v6.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5906) TestChangingEncoding failing sporadically in 0.94 build

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265671#comment-13265671
 ] 

Hudson commented on HBASE-5906:
---

Integrated in HBase-TRUNK-security #188 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/188/])
HBASE-5906 TestChangingEncoding failing sporadically in 0.94 build 
(Revision 1332320)

 Result = FAILURE
stack : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java


> TestChangingEncoding failing sporadically in 0.94 build
> ---
>
> Key: HBASE-5906
> URL: https://issues.apache.org/jira/browse/HBASE-5906
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
> Attachments: 5906.txt
>
>
> The test passes locally for me and Elliott but takes a long time to run.  
> Timeout is only two minutes for the test though.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5884) MapReduce package info has broken link to bulk-loads

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265672#comment-13265672
 ] 

Hudson commented on HBASE-5884:
---

Integrated in HBase-TRUNK-security #188 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/188/])
HBASE-5884 MapReduce package info has broken link to bulk-loads (Revision 
1332440)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/package-info.java


> MapReduce package info has broken link to bulk-loads
> 
>
> Key: HBASE-5884
> URL: https://issues.apache.org/jira/browse/HBASE-5884
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.96.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Trivial
> Fix For: 0.94.0, 0.96.0
>
> Attachments: doc_HBASE-5884.patch
>
>
> Bulk Loads link goes to an old link, which we have dropped recently.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265668#comment-13265668
 ] 

Hudson commented on HBASE-5548:
---

Integrated in HBase-TRUNK-security #188 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/188/])
HBASE-5548 Add ability to get a table in the shell (Revision 1332419)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/ruby/hbase/hbase.rb
* /hbase/trunk/src/main/ruby/hbase/table.rb
* /hbase/trunk/src/main/ruby/shell.rb
* /hbase/trunk/src/main/ruby/shell/commands.rb
* /hbase/trunk/src/main/ruby/shell/commands/count.rb
* /hbase/trunk/src/main/ruby/shell/commands/create.rb
* /hbase/trunk/src/main/ruby/shell/commands/delete.rb
* /hbase/trunk/src/main/ruby/shell/commands/deleteall.rb
* /hbase/trunk/src/main/ruby/shell/commands/get.rb
* /hbase/trunk/src/main/ruby/shell/commands/get_counter.rb
* /hbase/trunk/src/main/ruby/shell/commands/get_table.rb
* /hbase/trunk/src/main/ruby/shell/commands/incr.rb
* /hbase/trunk/src/main/ruby/shell/commands/put.rb
* /hbase/trunk/src/main/ruby/shell/commands/scan.rb
* /hbase/trunk/src/main/ruby/shell/commands/table_help.rb
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* /hbase/trunk/src/test/ruby/hbase/admin_test.rb


> Add ability to get a table in the shell
> ---
>
> Key: HBASE-5548
> URL: https://issues.apache.org/jira/browse/HBASE-5548
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.96.0
>
> Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, 
> ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch
>
>
> Currently, all the commands that operate on a table in the shell first have 
> to take the table as name as input. 
> There are two main considerations:
> * It is annoying to have to write the table name every time, when you should 
> just be able to get a reference to a table
> * the current implementation is very wasteful - it creates a new HTable for 
> each call (but reuses the connection since it uses the same configuration)
> We should be able to get a handle to a single HTable and then operate on that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5903) Detect the test classes without categories

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265670#comment-13265670
 ] 

Hudson commented on HBASE-5903:
---

Integrated in HBase-TRUNK-security #188 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/188/])
HBASE-5903 Detect the test classes without categories (Revision 1332260)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestCheckTestClasses.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java


> Detect the test classes without categories
> --
>
> Key: HBASE-5903
> URL: https://issues.apache.org/jira/browse/HBASE-5903
> Project: HBase
>  Issue Type: Improvement
>  Components: build, test
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5903.v3.patch, 5903v4.txt
>
>
> The tests are executed by category. When a test does not have a category, 
> it's not run on prebuild nor central build.
> This new test checks the test classess and list the ones without category. It 
> fails if it finds one. As it's a small test it will be executed on the 
> developper machine and will fail immediately on the central builds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265666#comment-13265666
 ] 

Hudson commented on HBASE-5908:
---

Integrated in HBase-TRUNK-security #188 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/188/])
HBASE-5908. TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses 
should not use append to corrupt the HLog. Contributed by Gregory Chanan. 
(Revision 1332495)

 Result = FAILURE
todd : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java


> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5879) Enable JMX metrics collection for the Thrift proxy

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265669#comment-13265669
 ] 

Hudson commented on HBASE-5879:
---

Integrated in HBase-TRUNK-security #188 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/188/])
HBASE-5879 Enable JMX metrics collection for the Thrift proxy (Revision 
1332450)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/bin/hbase-config.sh


> Enable JMX metrics collection for the Thrift proxy
> --
>
> Key: HBASE-5879
> URL: https://issues.apache.org/jira/browse/HBASE-5879
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5879_trunk.txt, D2955.1.patch
>
>
> We need to enable JMX on the Thrift proxy on a separate port different from 
> the JMX port used by regionserver. This is necessary for metrics collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5712) Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265667#comment-13265667
 ] 

Hudson commented on HBASE-5712:
---

Integrated in HBase-TRUNK-security #188 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/188/])
HBASE-5712 Parallelize load of .regioninfo files in diagnostic/repair 
portion of hbck (Revision 1332072)

 Result = FAILURE
jmhsieh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java


> Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
> ---
>
> Key: HBASE-5712
> URL: https://issues.apache.org/jira/browse/HBASE-5712
> Project: HBase
>  Issue Type: Improvement
>  Components: hbck
>Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
> Attachments: hbase-5712-90-v2.patch, hbase-5712-90.patch, 
> hbase-5712-v2.patch, hbase-5712.patch
>
>
> On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off 
> for 60s before attempting to read data from another datanode.  Portions of 
> the information gathered from hdfs (.regioninfo files) are loaded serially.  
> With HBase with clusters with 100's, or 1000's, or 1's regions 
> encountering these 60s delay blocks progress and can be very painful.  
> There is already some parallelization of portions of the hdfs information 
> load operations and the goal here is move the reading of .regioninfos into 
> the parallelized sections..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265648#comment-13265648
 ] 

ramkrishna.s.vasudevan commented on HBASE-5699:
---

Do we need to gurantee the HLog edits sequencing even with multiple WALs? Just 
referring to Stack's comment in 
https://issues.apache.org/jira/browse/HBASE-5782?focusedCommentId=13255344&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13255344

> Run with > 1 WAL in HRegionServer
> -
>
> Key: HBASE-5699
> URL: https://issues.apache.org/jira/browse/HBASE-5699
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Li Pi
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5909) SlabStats should be a daemon thread

[Jean-Daniel Cryans (JIRA)]

Jean-Daniel Cryans created HBASE-5909:
-

 Summary: SlabStats should be a daemon thread
 Key: HBASE-5909
 URL: https://issues.apache.org/jira/browse/HBASE-5909
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1
Reporter: Jean-Daniel Cryans
 Fix For: 0.96.0, 0.94.1


I had a hanging JVM on shutdown caused by:

{noformat}
"Slab Statistics #0" prio=5 tid=7fc0238bc800 nid=0x10dadf000 waiting on 
condition [10dade000]
   java.lang.Thread.State: TIMED_WAITING (parking)
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5840) Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing the old status

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265642#comment-13265642
 ] 

ramkrishna.s.vasudevan commented on HBASE-5840:
---

@Lars
Can i commit this to both versions? 0.94 and trunk?

> Open Region FAILED_OPEN doesn't clear the TaskMonitor Status, keeps showing 
> the old status
> --
>
> Key: HBASE-5840
> URL: https://issues.apache.org/jira/browse/HBASE-5840
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.94.0
>Reporter: Gopinathan A
>Assignee: rajeshbabu
> Fix For: 0.96.0, 0.94.1
>
> Attachments: HBASE-5840.patch, HBASE-5840_trunk.patch, 
> HBASE-5840_v2.patch
>
>
> TaskMonitor Status will not be cleared in case Regions FAILED_OPEN. This will 
> keeps showing old status.
> This will miss leads the user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265637#comment-13265637
 ] 

Hudson commented on HBASE-5908:
---

Integrated in HBase-0.92 #394 (See 
[https://builds.apache.org/job/HBase-0.92/394/])
HBASE-5908. TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses 
should not use append to corrupt the HLog. Contributed by Gregory Chanan. 
(Revision 1332493)

 Result = SUCCESS
todd : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java


> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-5155:
--

Release Note: 
This issue is an incompatible change.
If an HBase client with the changes for HBASE-5155 and a server (master) 
without the changes for HBASE-5155 is used, then the is_enabled (from HBase 
Shell) or isTableEnabled() (from HBaseAdmin) will return false though the table 
is already enabled as per the master.

If the HBase client does have the changes for HBASE-5155 and the server does 
not have the changes for HBASE-5155, then if we try to Enable a table then the 
client will hang.

The reason is because,
Prior to HBASE-5155 once the table is enabled the znode in the zookeeper 
created for the table is deleted.
After HBASE-5155 once the table is enabled the znode in the zookeeper created 
for the table is not deleted, whereas the same node is updated with the status 
ENABLED.

The client also expects the status of the znode in the zookeeper to be in the 
ENABLED state if the table has been enabled successfully.
The above changes makes the client behaviour incompatible if the client does 
not have this fix whereas the server has this fix.
If both the client and the server does not have this fix, then the behaviour is 
as expected.

  was:
This issue is an incompatible change.
If an HBase client with the changes for HBASE-5155 and a server (master) 
without the changes for HBASE-5155 is used, then the is_enabled (from HBase 
Shell) or isTableEnabled() (from HBaseAdmin) will return false though the table 
is already enabled as per the master.

If the HBase client does not have the changes for HBASE-5155 and the server has 
the changes for HBASE-5155, then if we try to Enable a table then the client 
will hang.

The reason is because,
Prior to HBASE-5155 once the table is enabled the znode in the zookeeper 
created for the table is deleted.
After HBASE-5155 once the table is enabled the znode in the zookeeper created 
for the table is not deleted, whereas the same node is updated with the status 
ENABLED.

The client also expects the status of the znode in the zookeeper to be in the 
ENABLED state if the table has been enabled successfully.
The above changes makes the client behaviour incompatible if the client does 
not have this fix whereas the server has this fix.
If both the client and the server does not have this fix, then the behaviour is 
as expected.


> ServerShutDownHandler And Disable/Delete should not happen parallely leading 
> to recreation of regions that were deleted
> ---
>
> Key: HBASE-5155
> URL: https://issues.apache.org/jira/browse/HBASE-5155
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.4
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.90.6
>
> Attachments: HBASE-5155_1.patch, HBASE-5155_2.patch, 
> HBASE-5155_3.patch, HBASE-5155_latest.patch, hbase-5155_6.patch
>
>
> ServerShutDownHandler and disable/delete table handler races.  This is not an 
> issue due to TM.
> -> A regionserver goes down.  In our cluster the regionserver holds lot of 
> regions.
> -> A region R1 has two daughters D1 and D2.
> -> The ServerShutdownHandler gets called and scans the META and gets all the 
> user regions
> -> Parallely a table is disabled. (No problem in this step).
> -> Delete table is done.
> -> The tables and its regions are deleted including R1, D1 and D2.. (So META 
> is cleaned)
> -> Now ServerShutdownhandler starts to processTheDeadRegion
> {code}
>  if (hri.isOffline() && hri.isSplit()) {
>   LOG.debug("Offlined and split region " + hri.getRegionNameAsString() +
> "; checking daughter presence");
>   fixupDaughters(result, assignmentManager, catalogTracker);
> {code}
> As part of fixUpDaughters as the daughers D1 and D2 is missing for R1 
> {code}
> if (isDaughterMissing(catalogTracker, daughter)) {
>   LOG.info("Fixup; missing daughter " + daughter.getRegionNameAsString());
>   MetaEditor.addDaughter(catalogTracker, daughter, null);
>   // TODO: Log WARN if the regiondir does not exist in the fs.  If its not
>   // there then something wonky about the split -- things will keep going
>   // but could be missing references to parent region.
>   // And assign it.
>   assignmentManager.assign(daughter, true);
> {code}
> we call assign of the daughers.  
> Now after this we again start with the below code.
> {code}
> if (processDeadRegion(e.getKey(), e.getValue(),
> this.services.getAssignmentManager(),
> this.server.getCatalogTracker())) {
>   this.services.getA

[jira] [Commented] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265633#comment-13265633
 ] 

ramkrishna.s.vasudevan commented on HBASE-5155:
---

@David
Updated the release notes. Thanks for your review.

> ServerShutDownHandler And Disable/Delete should not happen parallely leading 
> to recreation of regions that were deleted
> ---
>
> Key: HBASE-5155
> URL: https://issues.apache.org/jira/browse/HBASE-5155
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.4
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.90.6
>
> Attachments: HBASE-5155_1.patch, HBASE-5155_2.patch, 
> HBASE-5155_3.patch, HBASE-5155_latest.patch, hbase-5155_6.patch
>
>
> ServerShutDownHandler and disable/delete table handler races.  This is not an 
> issue due to TM.
> -> A regionserver goes down.  In our cluster the regionserver holds lot of 
> regions.
> -> A region R1 has two daughters D1 and D2.
> -> The ServerShutdownHandler gets called and scans the META and gets all the 
> user regions
> -> Parallely a table is disabled. (No problem in this step).
> -> Delete table is done.
> -> The tables and its regions are deleted including R1, D1 and D2.. (So META 
> is cleaned)
> -> Now ServerShutdownhandler starts to processTheDeadRegion
> {code}
>  if (hri.isOffline() && hri.isSplit()) {
>   LOG.debug("Offlined and split region " + hri.getRegionNameAsString() +
> "; checking daughter presence");
>   fixupDaughters(result, assignmentManager, catalogTracker);
> {code}
> As part of fixUpDaughters as the daughers D1 and D2 is missing for R1 
> {code}
> if (isDaughterMissing(catalogTracker, daughter)) {
>   LOG.info("Fixup; missing daughter " + daughter.getRegionNameAsString());
>   MetaEditor.addDaughter(catalogTracker, daughter, null);
>   // TODO: Log WARN if the regiondir does not exist in the fs.  If its not
>   // there then something wonky about the split -- things will keep going
>   // but could be missing references to parent region.
>   // And assign it.
>   assignmentManager.assign(daughter, true);
> {code}
> we call assign of the daughers.  
> Now after this we again start with the below code.
> {code}
> if (processDeadRegion(e.getKey(), e.getValue(),
> this.services.getAssignmentManager(),
> this.server.getCatalogTracker())) {
>   this.services.getAssignmentManager().assign(e.getKey(), true);
> {code}
> Now when the SSH scanned the META it had R1, D1 and D2.
> So as part of the above code D1 and D2 which where assigned by fixUpDaughters
> is again assigned by 
> {code}
> this.services.getAssignmentManager().assign(e.getKey(), true);
> {code}
> Thus leading to a zookeeper issue due to bad version and killing the master.
> The important part here is the regions that were deleted are recreated which 
> i think is more critical.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265627#comment-13265627
 ] 

Zhihong Yu commented on HBASE-5547:
---

@Jesse:
Do you want to attach patch so that Hadoop QA can run test suite ?

> Don't delete HFiles when in "backup mode"
> -
>
> Key: HBASE-5547
> URL: https://issues.apache.org/jira/browse/HBASE-5547
> Project: HBase
>  Issue Type: New Feature
>Reporter: Lars Hofhansl
>Assignee: Jesse Yates
>
> This came up in a discussion I had with Stack.
> It would be nice if HBase could be notified that a backup is in progress (via 
> a znode for example) and in that case either:
> 1. rename HFiles to be delete to .bck
> 2. rename the HFiles into a special directory
> 3. rename them to a general trash directory (which would not need to be tied 
> to backup mode).
> That way it should be able to get a consistent backup based on HFiles (HDFS 
> snapshots or hard links would be better options here, but we do not have 
> those).
> #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265620#comment-13265620
 ] 

Zhihong Yu commented on HBASE-5699:
---

Currently we maintain one sequence number per region per HLog. From append():
{code}
  this.lastSeqWritten.putIfAbsent(regionInfo.getEncodedNameAsBytes(),
Long.valueOf(seqNum));
{code}
If WALEdit's from a particular region can spread across multiple streams, 
accounting would be more complex.

> Run with > 1 WAL in HRegionServer
> -
>
> Key: HBASE-5699
> URL: https://issues.apache.org/jira/browse/HBASE-5699
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Li Pi
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265618#comment-13265618
 ] 

Hudson commented on HBASE-5908:
---

Integrated in HBase-0.94 #164 (See 
[https://builds.apache.org/job/HBase-0.94/164/])
HBASE-5908. TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses 
should not use append to corrupt the HLog. Contributed by Gregory Chanan. 
(Revision 1332494)

 Result = FAILURE
todd : 
Files : 
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java


> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265615#comment-13265615
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/#review7411
---



src/main/resources/hbase-webapps/master/table.jsp


Can this computation be moved somewhere so that it can be reused ?


- Ted


On 2012-05-01 01:29:14, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4463/
bq.  ---
bq.  
bq.  (Updated 2012-05-01 01:29:14)
bq.  
bq.  
bq.  Review request for hbase and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Adds PB-based calls replacing HMasterRegionInterface.
bq.  
bq.  There are some temporary hacks, e.g. converting PB-based ServerLoad to 
existing HServerLoad so I didn't need to convert ClusterStatus (which brings in 
a lot of other changes).  That will be cleaned up in HBASE-5445.
bq.  
bq.  
bq.  This addresses bug HBASE-5444.
bq.  https://issues.apache.org/jira/browse/HBASE-5444
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 
fd97830 
bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 
f56127d 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java 
be63838 
bq.
src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
ebffad6 
bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
bq.src/main/protobuf/hbase.proto 12e6053 
bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 
f8029ba 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 
bq.  
bq.  Diff: https://reviews.apache.org/r/4463/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran jenkins job, all unit tests passed.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



> Add PB-based calls to HMasterRegionInterface
> 
>
> Key: HBASE-5444
> URL: https://issues.apache.org/jira/browse/HBASE-5444
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: Gregory Chanan
> Attachments: HBASE-5444-v6-trunk.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265614#comment-13265614
 ] 

Lars Hofhansl commented on HBASE-5897:
--

+1 on either v2 or v3, btw.




> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3.txt, 
> hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265613#comment-13265613
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/
---

(Updated 2012-05-01 02:42:34.215505)


Review request for hbase, Michael Stack and Lars Hofhansl.


Changes
---

 - TableTracker is now an abstract class that the catalogjanitor and rs are
both using to register for tracking a table.
 - Adding monitoring in admin to check for cj to join archiving (before was 
missed) and including test
 - Slight cleanup from Ted's comments.


Summary
---

Essentially, whenever an hfile would be deleted, it is instead moved to the 
archive directory. In this impl, the archive directory is on a per table basis, 
but defaults to '.archive'. Removing hfiles occurs in three places - 
compaction, merge and catalog janitor. The former and two latter are distinctly 
different code paths, but but did pull out some similarities. The latter two 
end up calling the same method, so there should be a reasonable amount of 
overlap.

Implementation wise: 
Updated the HMasterInterface to pass the calls onto the zookeeper.
Added a zk listener to handle updates from the master to the RS to backup.
Added a utility for removing files and finding archive directories
Added tests for the regionserver and catalogjanitor approaches.
Added creation of manager in regionserver.


This addresses bug HBASE-5547.
https://issues.apache.org/jira/browse/HBASE-5547


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java 
PRE-CREATION 
  
src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java
 PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 
  src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
6884d53 
  src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 
ea12da4 
  src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
  src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f 
  src/main/resources/hbase-default.xml f54b345 
  src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 
  src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java
 PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 
7d02759 
  src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/4633/diff


Testing
---

Added two tests for the separate cases - archiving via the regionserver and for 
the catalog tracker. Former runs in a mini cluster and also touches the changes 
to HMasterInterface and zookeeper.


Thanks,

Jesse



> Don't delete HFiles when in "backup mode"
> -
>
> Key: HBASE-5547
> URL: https://issues.apache.org/jira/browse/HBASE-5547
> Project: HBase
>  Issue Type: New Feature
>Reporter: Lars Hofhansl
>Assignee: Jesse Yates
>
> This came up in a discussion I had with Stack.
> It would be nice if HBase could be notified that a backup is in progress (via 
> a znode for example) and in that case either:
> 1. rename HFiles to be delete to .bck
> 2. rename the HFiles into a special directory
> 3. rename them to a general trash directory (which would not need to be tied 
> to backup mode).
> That way it should be able to get a consistent backup based on HFiles (HDFS 
> snapshots or hard links would be better options here, but we do not have 
> those).
> #1 makes cleanup a bit harder.

-

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265612#comment-13265612
 ] 

Hudson commented on HBASE-5908:
---

Integrated in HBase-TRUNK #2830 (See 
[https://builds.apache.org/job/HBase-TRUNK/2830/])
HBASE-5908. TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses 
should not use append to corrupt the HLog. Contributed by Gregory Chanan. 
(Revision 1332495)

 Result = FAILURE
todd : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java


> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265609#comment-13265609
 ] 

jirapos...@reviews.apache.org commented on HBASE-5547:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review7410
---



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java


agreed. I'll try to find something better.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java


zk add/remove conditions where you need to keep re-adding watches. figure 
we do the check for the table at the same time.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java


removed the creation of the tracker in favor of just passing in one - its 
really light weight and this should remove some of the issues around it; in 
short, yes.



src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java


switching this mechanism to allow more fine grained listening by table 
trackers.


- Jesse


On 2012-04-26 00:24:36, Jesse Yates wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4633/
bq.  ---
bq.  
bq.  (Updated 2012-04-26 00:24:36)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack and Lars Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Essentially, whenever an hfile would be deleted, it is instead moved to 
the archive directory. In this impl, the archive directory is on a per table 
basis, but defaults to '.archive'. Removing hfiles occurs in three places - 
compaction, merge and catalog janitor. The former and two latter are distinctly 
different code paths, but but did pull out some similarities. The latter two 
end up calling the same method, so there should be a reasonable amount of 
overlap.
bq.  
bq.  Implementation wise: 
bq.  Updated the HMasterInterface to pass the calls onto the zookeeper.
bq.  Added a zk listener to handle updates from the master to the RS to 
backup.
bq.  Added a utility for removing files and finding archive directories
bq.  Added tests for the regionserver and catalogjanitor approaches.
bq.  Added creation of manager in regionserver.
bq.  
bq.  
bq.  This addresses bug HBASE-5547.
bq.  https://issues.apache.org/jira/browse/HBASE-5547
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 
bq.src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
61a5988 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerHFileTableArchiveTracker.java
 PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
6884d53 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 
ea12da4 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e 
bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 
4fc105f 
bq.src/main/resources/hbase-default.xml f54b345 
bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 
a59e152 
bq.src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 
cedf31e 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java
 PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 
7d02759 
bq.
src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4633/diff
bq.  
bq.  
bq.  Testing
bq.  --

[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265606#comment-13265606
 ] 

Zhihong Yu commented on HBASE-5699:
---

bq. to one "HLog" object, which might have more than one underlying stream.
The above can be a (sub-)task by itself.


> Run with > 1 WAL in HRegionServer
> -
>
> Key: HBASE-5699
> URL: https://issues.apache.org/jira/browse/HBASE-5699
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Li Pi
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265604#comment-13265604
 ] 

Hadoop QA commented on HBASE-5444:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12525149/HBASE-5444-v6-trunk.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 18 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 3 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.client.TestShell
  org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1702//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1702//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1702//console

This message is automatically generated.

> Add PB-based calls to HMasterRegionInterface
> 
>
> Key: HBASE-5444
> URL: https://issues.apache.org/jira/browse/HBASE-5444
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: Gregory Chanan
> Attachments: HBASE-5444-v6-trunk.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265596#comment-13265596
 ] 

Todd Lipcon commented on HBASE-5699:


bq. Currently each HRegion has reference to the HLog it uses. If requests can 
be freely redirected to the HLog instance having fewer outstanding requests, 
the reference would be to that of the region server.

Sorry, I should be less free-wheeling with my terminology. My thought was that 
there is still a single "HLog" class, but underneath it would be multiple 
"SequenceFileLogWriters", most likely. Though maybe the correct implementation 
is to make HLog an interface, and then have a MultiHLog which wraps N other 
HLogs or something. Either way, any region would only have a reference to one 
"HLog" object, which might have more than one underlying stream.

> Run with > 1 WAL in HRegionServer
> -
>
> Key: HBASE-5699
> URL: https://issues.apache.org/jira/browse/HBASE-5699
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Li Pi
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Todd Lipcon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-5908:
---

   Resolution: Fixed
Fix Version/s: 0.96.0
   0.94.0
   0.92.2
   0.90.7
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[Gregory Chanan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan updated HBASE-5444:
--

Status: Patch Available  (was: Open)

> Add PB-based calls to HMasterRegionInterface
> 
>
> Key: HBASE-5444
> URL: https://issues.apache.org/jira/browse/HBASE-5444
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: Gregory Chanan
> Attachments: HBASE-5444-v6-trunk.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[Gregory Chanan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan updated HBASE-5444:
--

Attachment: HBASE-5444-v6-trunk.patch

> Add PB-based calls to HMasterRegionInterface
> 
>
> Key: HBASE-5444
> URL: https://issues.apache.org/jira/browse/HBASE-5444
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: Gregory Chanan
> Attachments: HBASE-5444-v6-trunk.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265585#comment-13265585
 ] 

jirapos...@reviews.apache.org commented on HBASE-5444:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4463/
---

(Updated 2012-05-01 01:29:14.986766)


Review request for hbase and Michael Stack.


Summary
---

Adds PB-based calls replacing HMasterRegionInterface.

There are some temporary hacks, e.g. converting PB-based ServerLoad to existing 
HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of 
other changes).  That will be cleaned up in HBASE-5445.


This addresses bug HBASE-5444.
https://issues.apache.org/jira/browse/HBASE-5444


Diffs
-

  src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 
69434f7 
  src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
3c7c091 
  src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b 
  src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 
  src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b 
  src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 
  src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 
  src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 
  src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 
  src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
efcf74d 
  
src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
 PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 
  src/main/protobuf/RegionServerStatus.proto PRE-CREATION 
  src/main/protobuf/hbase.proto 12e6053 
  src/main/resources/hbase-webapps/master/table.jsp 3ef1190 
  src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb 
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 
d039be3 
  src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 
36046f8 
  src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 
  src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba 
  
src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
 e99d251 

Diff: https://reviews.apache.org/r/4463/diff


Testing
---

Ran jenkins job, all unit tests passed.


Thanks,

Gregory



> Add PB-based calls to HMasterRegionInterface
> 
>
> Key: HBASE-5444
> URL: https://issues.apache.org/jira/browse/HBASE-5444
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: Gregory Chanan
> Attachments: HBASE-5444-v6-trunk.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265582#comment-13265582
 ] 

Todd Lipcon commented on HBASE-5908:


I'll commit this momentarily to 0.90, 0.92, 0.94, and trunk (whew)

> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Gregory Chanan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265580#comment-13265580
 ] 

Gregory Chanan commented on HBASE-5908:
---

Those tests fail for me locally even without patch applied.

> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265569#comment-13265569
 ] 

Hadoop QA commented on HBASE-5897:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12525142/5897-v3.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.master.TestSplitLogManager
  org.apache.hadoop.hbase.client.TestShell

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1701//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1701//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1701//console

This message is automatically generated.

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3.txt, 
> hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265567#comment-13265567
 ] 

Jonathan Hsieh commented on HBASE-5699:
---

The argument here is mostly aimed at read latency, but a similar idea could be 
used for write latency as well.

> Run with > 1 WAL in HRegionServer
> -
>
> Key: HBASE-5699
> URL: https://issues.apache.org/jira/browse/HBASE-5699
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Li Pi
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265565#comment-13265565
 ] 

Jonathan Hsieh commented on HBASE-5699:
---

Part of the motivation for multiple wals can be found in this tech talk: (most 
relavent to HBase is backup requests, starting slide 39)

http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/people/jeff/Berkeley-Latency-Mar2012.pdf

> Run with > 1 WAL in HRegionServer
> -
>
> Key: HBASE-5699
> URL: https://issues.apache.org/jira/browse/HBASE-5699
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Li Pi
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1326#comment-1326
 ] 

Hadoop QA commented on HBASE-5897:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12525137/5897-v2.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.client.TestShell

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1700//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1700//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1700//console

This message is automatically generated.

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3.txt, 
> hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5860) splitlogmanager should not unnecessarily resubmit tasks when zk unavailable

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265544#comment-13265544
 ] 

Hadoop QA commented on HBASE-5860:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12525131/0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 4 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.client.TestShell

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1699//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1699//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1699//console

This message is automatically generated.

> splitlogmanager should not unnecessarily resubmit tasks when zk unavailable
> ---
>
> Key: HBASE-5860
> URL: https://issues.apache.org/jira/browse/HBASE-5860
> Project: HBase
>  Issue Type: Improvement
>Reporter: Prakash Khemani
>Assignee: Prakash Khemani
> Attachments: 
> 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch, 
> 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch
>
>
> (Doesn't really impact the run time or correctness of log splitting)
> say the master has lost connection to zk. splitlogmanager's timeoutmanager 
> will realize that all the tasks that were submitted are still unassigned. It 
> will resubmit those tasks (i.e. create dummy znodes)
> splitlogmanager should realze that the tasks are unassigned but their znodes 
> have not been created.
> 012-04-20 13:11:20,516 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> dead splitlog worker msgstore295.snc4.facebook.com,60020,1334948757026
> 2012-04-20 13:11:20,517 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> Scheduling batch of logs to split
> 2012-04-20 13:11:20,517 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> started splitting logs in 
> [hdfs://msgstore215.snc4.facebook.com:9000/MSGSTORE215-SNC4-HBASE/.logs/msgstore295.snc4.facebook.com,60020,1334948757026-splitting]
> 2012-04-20 13:11:20,565 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
> connection to server msgstore235.snc4.facebook.com/10.30.222.186:2181
> 2012-04-20 13:11:20,566 INFO org.apache.zookeeper.ClientCnxn: Socket 
> connection established to msgstore235.snc4.facebook.com/10.30.222.186:2181, 
> initiating session
> 2012-04-20 13:11:20,575 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> total tasks = 4 unassigned = 4
> 2012-04-20 13:11:20,576 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,577 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x36ccb0f8010002, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x136ccb0f489, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951586677
>  retry=3
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2

[jira] [Updated] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5897:
-

Attachment: 5897-v3.txt

How about this one. Just removes the precondition checks, applies prePut edit 
edits before the family edits. And removes an unused variable.

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3.txt, 
> hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Gregory Chanan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265530#comment-13265530
 ] 

Gregory Chanan commented on HBASE-5908:
---

@Ted:
Good idea, I'll do that from now on.

> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265529#comment-13265529
 ] 

Zhihong Yu commented on HBASE-5548:
---

@Jesse:
I checked the two outstanding Hadoop QA jobs around 23:27 - they were not for 
this JIRA.

> Add ability to get a table in the shell
> ---
>
> Key: HBASE-5548
> URL: https://issues.apache.org/jira/browse/HBASE-5548
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.96.0
>
> Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, 
> ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch
>
>
> Currently, all the commands that operate on a table in the shell first have 
> to take the table as name as input. 
> There are two main considerations:
> * It is annoying to have to write the table name every time, when you should 
> just be able to get a reference to a table
> * the current implementation is very wasteful - it creates a new HTable for 
> each call (but reuses the connection since it uses the same configuration)
> We should be able to get a handle to a single HTable and then operate on that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265528#comment-13265528
 ] 

Hadoop QA commented on HBASE-5908:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12525130/HBASE-5908-trunk.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.client.TestShell
  org.apache.hadoop.hbase.master.TestAssignmentManager

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1698//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1698//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1698//console

This message is automatically generated.

> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265527#comment-13265527
 ] 

Zhihong Yu commented on HBASE-5908:
---

@Gregory:
I would suggest referencing other JIRAs by their names only, such as 
HADOOP-8230.

This way we would easily see whether the JIRA has been resolved.

> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265521#comment-13265521
 ] 

Zhihong Yu commented on HBASE-5699:
---

Trying to understand the implication of Todd's suggestion above.
Currently each HRegion has reference to the HLog it uses. If requests can be 
freely redirected to the HLog instance having fewer outstanding requests, the 
reference would be to that of the region server.
This means additional logic on region server for dispatching the write requests.

> Run with > 1 WAL in HRegionServer
> -
>
> Key: HBASE-5699
> URL: https://issues.apache.org/jira/browse/HBASE-5699
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Li Pi
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell

[Jesse Yates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265519#comment-13265519
 ] 

Jesse Yates commented on HBASE-5548:


yeah, was waiting for the QA run on it... maybe we jumped the gun a little 
there (though TestShell has been always a little funky on my machine). Looking 
into the issue now.

> Add ability to get a table in the shell
> ---
>
> Key: HBASE-5548
> URL: https://issues.apache.org/jira/browse/HBASE-5548
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.96.0
>
> Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, 
> ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch
>
>
> Currently, all the commands that operate on a table in the shell first have 
> to take the table as name as input. 
> There are two main considerations:
> * It is annoying to have to write the table name every time, when you should 
> just be able to get a reference to a table
> * the current implementation is very wasteful - it creates a new HTable for 
> each call (but reuses the connection since it uses the same configuration)
> We should be able to get a handle to a single HTable and then operate on that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265518#comment-13265518
 ] 

Zhihong Yu commented on HBASE-5897:
---

Sorry, Lars.
Attaching jstack was to show that patch v1 needed a little refinement.
Then I went to talk to my manager, and ...

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, 5897-v2.txt, hbase-5897.txt, 
> testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5879) Enable JMX metrics collection for the Thrift proxy

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265515#comment-13265515
 ] 

Hudson commented on HBASE-5879:
---

Integrated in HBase-TRUNK #2829 (See 
[https://builds.apache.org/job/HBase-TRUNK/2829/])
HBASE-5879 Enable JMX metrics collection for the Thrift proxy (Revision 
1332450)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/bin/hbase-config.sh


> Enable JMX metrics collection for the Thrift proxy
> --
>
> Key: HBASE-5879
> URL: https://issues.apache.org/jira/browse/HBASE-5879
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5879_trunk.txt, D2955.1.patch
>
>
> We need to enable JMX on the Thrift proxy on a separate port different from 
> the JMX port used by regionserver. This is necessary for metrics collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265517#comment-13265517
 ] 

Lars Hofhansl commented on HBASE-5897:
--

You beat me to it Ted :)

I was also going to suggest to remove the new precondition checks. In both 
cases nothing incorrect will happen (if the prePut skips or the Put has 
writeToWal==false, the WAL is not applied), but we're changing the coprocessor 
protocol slightly.


> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, 5897-v2.txt, hbase-5897.txt, 
> testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5897:
--

Attachment: 5897-v2.txt

Patch v2 passes TestRegionServerCoprocessorExceptionWithRemove

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, 5897-v2.txt, hbase-5897.txt, 
> testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5860) splitlogmanager should not unnecessarily resubmit tasks when zk unavailable

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5860:
--

Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

> splitlogmanager should not unnecessarily resubmit tasks when zk unavailable
> ---
>
> Key: HBASE-5860
> URL: https://issues.apache.org/jira/browse/HBASE-5860
> Project: HBase
>  Issue Type: Improvement
>Reporter: Prakash Khemani
>Assignee: Prakash Khemani
> Attachments: 
> 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch, 
> 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch
>
>
> (Doesn't really impact the run time or correctness of log splitting)
> say the master has lost connection to zk. splitlogmanager's timeoutmanager 
> will realize that all the tasks that were submitted are still unassigned. It 
> will resubmit those tasks (i.e. create dummy znodes)
> splitlogmanager should realze that the tasks are unassigned but their znodes 
> have not been created.
> 012-04-20 13:11:20,516 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> dead splitlog worker msgstore295.snc4.facebook.com,60020,1334948757026
> 2012-04-20 13:11:20,517 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> Scheduling batch of logs to split
> 2012-04-20 13:11:20,517 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> started splitting logs in 
> [hdfs://msgstore215.snc4.facebook.com:9000/MSGSTORE215-SNC4-HBASE/.logs/msgstore295.snc4.facebook.com,60020,1334948757026-splitting]
> 2012-04-20 13:11:20,565 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
> connection to server msgstore235.snc4.facebook.com/10.30.222.186:2181
> 2012-04-20 13:11:20,566 INFO org.apache.zookeeper.ClientCnxn: Socket 
> connection established to msgstore235.snc4.facebook.com/10.30.222.186:2181, 
> initiating session
> 2012-04-20 13:11:20,575 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> total tasks = 4 unassigned = 4
> 2012-04-20 13:11:20,576 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,577 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x36ccb0f8010002, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x136ccb0f489, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951586677
>  retry=3
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951920332
>  retry=3

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265507#comment-13265507
 ] 

Lars Hofhansl commented on HBASE-5897:
--

I see. The call to doPrePut should be the try/catch block, so that 
closeRegionOperation is executed in case it throws an exception.

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, hbase-5897.txt, 
> testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265503#comment-13265503
 ] 

Lars Hofhansl commented on HBASE-5897:
--

That's a huge stack dump. What sticks to me are these:
{code}
at 
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:711)
at 
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove.teardownAfterClass(TestRegionServerCoprocessorExceptionWithRemove.java:77)
{code}
and 
{code}
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(Reen
trantReadWriteLock.java:807)
at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:929
)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:879)
- locked <7934dcbd0> (a java.lang.Object)
at 
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:121)
{code}

On the face of it, it does seem unrelated.

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, hbase-5897.txt, 
> testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HBASE-5548) Add ability to get a table in the shell

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu reopened HBASE-5548:
---


I don't see Hadoop QA report on this JIRA.
I can reproduce the following test failure:
{code}
  testRunShellTests(org.apache.hadoop.hbase.client.TestShell): (RuntimeError) 
Shell unit tests failed. Check output file for details.
{code}

> Add ability to get a table in the shell
> ---
>
> Key: HBASE-5548
> URL: https://issues.apache.org/jira/browse/HBASE-5548
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.96.0
>
> Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, 
> ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch
>
>
> Currently, all the commands that operate on a table in the shell first have 
> to take the table as name as input. 
> There are two main considerations:
> * It is annoying to have to write the table name every time, when you should 
> just be able to get a reference to a table
> * the current implementation is very wasteful - it creates a new HTable for 
> each call (but reuses the connection since it uses the same configuration)
> We should be able to get a handle to a single HTable and then operate on that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5884) MapReduce package info has broken link to bulk-loads

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265493#comment-13265493
 ] 

Hudson commented on HBASE-5884:
---

Integrated in HBase-0.94 #163 (See 
[https://builds.apache.org/job/HBase-0.94/163/])
HBASE-5884 MapReduce package info has broken link to bulk-loads (Revision 
1332441)

 Result = FAILURE
stack : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/package-info.java


> MapReduce package info has broken link to bulk-loads
> 
>
> Key: HBASE-5884
> URL: https://issues.apache.org/jira/browse/HBASE-5884
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.96.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Trivial
> Fix For: 0.94.0, 0.96.0
>
> Attachments: doc_HBASE-5884.patch
>
>
> Bulk Loads link goes to an old link, which we have dropped recently.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5860) splitlogmanager should not unnecessarily resubmit tasks when zk unavailable

[Nicolas Spiegelberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265491#comment-13265491
 ] 

Nicolas Spiegelberg commented on HBASE-5860:


I guess changing the retries to 0 should also fix the HBASE-5890 problem as 
well?  We shouldn't get a NODEEXISTS return for the RESCAN because we create it 
as EPHEMERAL_SEQUENTIAL.

> splitlogmanager should not unnecessarily resubmit tasks when zk unavailable
> ---
>
> Key: HBASE-5860
> URL: https://issues.apache.org/jira/browse/HBASE-5860
> Project: HBase
>  Issue Type: Improvement
>Reporter: Prakash Khemani
>Assignee: Prakash Khemani
> Attachments: 
> 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch, 
> 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch
>
>
> (Doesn't really impact the run time or correctness of log splitting)
> say the master has lost connection to zk. splitlogmanager's timeoutmanager 
> will realize that all the tasks that were submitted are still unassigned. It 
> will resubmit those tasks (i.e. create dummy znodes)
> splitlogmanager should realze that the tasks are unassigned but their znodes 
> have not been created.
> 012-04-20 13:11:20,516 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> dead splitlog worker msgstore295.snc4.facebook.com,60020,1334948757026
> 2012-04-20 13:11:20,517 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> Scheduling batch of logs to split
> 2012-04-20 13:11:20,517 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> started splitting logs in 
> [hdfs://msgstore215.snc4.facebook.com:9000/MSGSTORE215-SNC4-HBASE/.logs/msgstore295.snc4.facebook.com,60020,1334948757026-splitting]
> 2012-04-20 13:11:20,565 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
> connection to server msgstore235.snc4.facebook.com/10.30.222.186:2181
> 2012-04-20 13:11:20,566 INFO org.apache.zookeeper.ClientCnxn: Socket 
> connection established to msgstore235.snc4.facebook.com/10.30.222.186:2181, 
> initiating session
> 2012-04-20 13:11:20,575 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> total tasks = 4 unassigned = 4
> 2012-04-20 13:11:20,576 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,577 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x36ccb0f8010002, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x136ccb0f489, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951586677
>  retry=3
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951920332
>  retry=3

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265485#comment-13265485
 ] 

Todd Lipcon commented on HBASE-5908:


+1 pending Jenkins

> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Todd Lipcon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-5908:
---

Status: Patch Available  (was: Open)

> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5860) splitlogmanager should not unnecessarily resubmit tasks when zk unavailable

[Prakash Khemani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Khemani updated HBASE-5860:
---

Attachment: 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch

Nicolas's feedback applied.

also reduced the RESCAN retries to 0.

> splitlogmanager should not unnecessarily resubmit tasks when zk unavailable
> ---
>
> Key: HBASE-5860
> URL: https://issues.apache.org/jira/browse/HBASE-5860
> Project: HBase
>  Issue Type: Improvement
>Reporter: Prakash Khemani
>Assignee: Prakash Khemani
> Attachments: 
> 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch, 
> 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch
>
>
> (Doesn't really impact the run time or correctness of log splitting)
> say the master has lost connection to zk. splitlogmanager's timeoutmanager 
> will realize that all the tasks that were submitted are still unassigned. It 
> will resubmit those tasks (i.e. create dummy znodes)
> splitlogmanager should realze that the tasks are unassigned but their znodes 
> have not been created.
> 012-04-20 13:11:20,516 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> dead splitlog worker msgstore295.snc4.facebook.com,60020,1334948757026
> 2012-04-20 13:11:20,517 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> Scheduling batch of logs to split
> 2012-04-20 13:11:20,517 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> started splitting logs in 
> [hdfs://msgstore215.snc4.facebook.com:9000/MSGSTORE215-SNC4-HBASE/.logs/msgstore295.snc4.facebook.com,60020,1334948757026-splitting]
> 2012-04-20 13:11:20,565 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
> connection to server msgstore235.snc4.facebook.com/10.30.222.186:2181
> 2012-04-20 13:11:20,566 INFO org.apache.zookeeper.ClientCnxn: Socket 
> connection established to msgstore235.snc4.facebook.com/10.30.222.186:2181, 
> initiating session
> 2012-04-20 13:11:20,575 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> total tasks = 4 unassigned = 4
> 2012-04-20 13:11:20,576 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,577 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x36ccb0f8010002, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x136ccb0f489, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951586677
>  retry=3
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951920332
>  retry=3

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Gregory Chanan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan updated HBASE-5908:
--

Attachment: HBASE-5908-trunk.patch

> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
> append to corrupt the HLog
> 
>
> Key: HBASE-5908
> URL: https://issues.apache.org/jira/browse/HBASE-5908
> Project: HBase
>  Issue Type: Bug
>  Components: test, wal
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5908-trunk.patch
>
>
> TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against 
> a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230
> The failure:
> "java.io.IOException: Append is not supported. Please see the 
> dfs.support.append configuration parameter."
> Instead of using append, we can probably just:
> - copy over the contents to a new file
> - append the garbage to the new file
> - copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5907) enhance HLog pretty printer to print additional useful stats

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265483#comment-13265483
 ] 

Phabricator commented on HBASE-5907:


Kannan has added CCs to the revision "[jira] [HBASE-5907] [89-fb] enhance HLog 
pretty printer to print additional useful stats".
Added CCs: HBase Diffs Facebook Group

REVISION DETAIL
  https://reviews.facebook.net/D2979

BRANCH
  svn


> enhance HLog pretty printer to print additional useful stats
> 
>
> Key: HBASE-5907
> URL: https://issues.apache.org/jira/browse/HBASE-5907
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Priority: Minor
> Attachments: D2979.1.patch, D2979.2.patch
>
>
> It would be useful for analysis purposes to enhance the HLog pretty printer 
> to optionally print a bunch of additional stats such as:
> 1) # of txns
> 2) # of KVs updated
> 3) avg size of txns
> 4) avg size of KVs
> 5) avg # of KVs written per txn
> 5) unique CF signatures involved in put/delete operatons; and breakdown of 
> some of the above metrics by these signatures, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5884) MapReduce package info has broken link to bulk-loads

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265476#comment-13265476
 ] 

Hudson commented on HBASE-5884:
---

Integrated in HBase-TRUNK #2828 (See 
[https://builds.apache.org/job/HBase-TRUNK/2828/])
HBASE-5884 MapReduce package info has broken link to bulk-loads (Revision 
1332440)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/package-info.java


> MapReduce package info has broken link to bulk-loads
> 
>
> Key: HBASE-5884
> URL: https://issues.apache.org/jira/browse/HBASE-5884
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.96.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Trivial
> Fix For: 0.94.0, 0.96.0
>
> Attachments: doc_HBASE-5884.patch
>
>
> Bulk Loads link goes to an old link, which we have dropped recently.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265475#comment-13265475
 ] 

Hudson commented on HBASE-5548:
---

Integrated in HBase-TRUNK #2828 (See 
[https://builds.apache.org/job/HBase-TRUNK/2828/])
HBASE-5548 Add ability to get a table in the shell (Revision 1332419)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/ruby/hbase/hbase.rb
* /hbase/trunk/src/main/ruby/hbase/table.rb
* /hbase/trunk/src/main/ruby/shell.rb
* /hbase/trunk/src/main/ruby/shell/commands.rb
* /hbase/trunk/src/main/ruby/shell/commands/count.rb
* /hbase/trunk/src/main/ruby/shell/commands/create.rb
* /hbase/trunk/src/main/ruby/shell/commands/delete.rb
* /hbase/trunk/src/main/ruby/shell/commands/deleteall.rb
* /hbase/trunk/src/main/ruby/shell/commands/get.rb
* /hbase/trunk/src/main/ruby/shell/commands/get_counter.rb
* /hbase/trunk/src/main/ruby/shell/commands/get_table.rb
* /hbase/trunk/src/main/ruby/shell/commands/incr.rb
* /hbase/trunk/src/main/ruby/shell/commands/put.rb
* /hbase/trunk/src/main/ruby/shell/commands/scan.rb
* /hbase/trunk/src/main/ruby/shell/commands/table_help.rb
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* /hbase/trunk/src/test/ruby/hbase/admin_test.rb


> Add ability to get a table in the shell
> ---
>
> Key: HBASE-5548
> URL: https://issues.apache.org/jira/browse/HBASE-5548
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.96.0
>
> Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, 
> ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch
>
>
> Currently, all the commands that operate on a table in the shell first have 
> to take the table as name as input. 
> There are two main considerations:
> * It is annoying to have to write the table name every time, when you should 
> just be able to get a reference to a table
> * the current implementation is very wasteful - it creates a new HTable for 
> each call (but reuses the connection since it uses the same configuration)
> We should be able to get a handle to a single HTable and then operate on that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5888) Clover profile in build

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265477#comment-13265477
 ] 

Hadoop QA commented on HBASE-5888:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12525118/hbase-clover_v2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.replication.TestReplication
  org.apache.hadoop.hbase.client.TestShell

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1696//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1696//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1696//console

This message is automatically generated.

> Clover profile in build
> ---
>
> Key: HBASE-5888
> URL: https://issues.apache.org/jira/browse/HBASE-5888
> Project: HBase
>  Issue Type: Improvement
>  Components: build, test
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-clover_v1.patch, hbase-clover_v2.patch
>
>
> Clover is disabled right now. I would like to add a profile that enables 
> clover reports. We can also backport this to 0.92, and 0.94, since we are 
> also interested in test coverage for those branches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5907) enhance HLog pretty printer to print additional useful stats

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265474#comment-13265474
 ] 

Phabricator commented on HBASE-5907:


Liyin has accepted the revision "[jira] [HBASE-5907] [89-fb] enhance HLog 
pretty printer to print additional useful stats".

  LGTM ! Thanks Kannan !

REVISION DETAIL
  https://reviews.facebook.net/D2979

BRANCH
  svn


> enhance HLog pretty printer to print additional useful stats
> 
>
> Key: HBASE-5907
> URL: https://issues.apache.org/jira/browse/HBASE-5907
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Priority: Minor
> Attachments: D2979.1.patch, D2979.2.patch
>
>
> It would be useful for analysis purposes to enhance the HLog pretty printer 
> to optionally print a bunch of additional stats such as:
> 1) # of txns
> 2) # of KVs updated
> 3) avg size of txns
> 4) avg size of KVs
> 5) avg # of KVs written per txn
> 5) unique CF signatures involved in put/delete operatons; and breakdown of 
> some of the above metrics by these signatures, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5907) enhance HLog pretty printer to print additional useful stats

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5907:
---

Attachment: D2979.2.patch

Kannan updated the revision "[jira] [HBASE-5907] [89-fb] enhance HLog pretty 
printer to print additional useful stats".
Reviewers: Liyin, JIRA

  remove unused imports


REVISION DETAIL
  https://reviews.facebook.net/D2979

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
  src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java


> enhance HLog pretty printer to print additional useful stats
> 
>
> Key: HBASE-5907
> URL: https://issues.apache.org/jira/browse/HBASE-5907
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Priority: Minor
> Attachments: D2979.1.patch, D2979.2.patch
>
>
> It would be useful for analysis purposes to enhance the HLog pretty printer 
> to optionally print a bunch of additional stats such as:
> 1) # of txns
> 2) # of KVs updated
> 3) avg size of txns
> 4) avg size of KVs
> 5) avg # of KVs written per txn
> 5) unique CF signatures involved in put/delete operatons; and breakdown of 
> some of the above metrics by these signatures, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5907) enhance HLog pretty printer to print additional useful stats

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5907:
---

Attachment: D2979.1.patch

Kannan requested code review of "[jira] [HBASE-5907] [89-fb] enhance HLog 
pretty printer to print additional useful stats".
Reviewers: Liyin, JIRA

  It would be useful for analysis purposes to enhance the HLog pretty printer 
to optionally print a bunch of additional stats such as:

  1) # of txns
  2) # of KVs updated
  3) avg size of txns
  4) avg size of KVs
  5) avg # of KVs written per txn
  5) unique CF signatures involved in put/delete operatons; and breakdown of 
some of the above metrics by these signatures, etc.

TEST PLAN
  unit tests

REVISION DETAIL
  https://reviews.facebook.net/D2979

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
  src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/6783/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.


> enhance HLog pretty printer to print additional useful stats
> 
>
> Key: HBASE-5907
> URL: https://issues.apache.org/jira/browse/HBASE-5907
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Priority: Minor
> Attachments: D2979.1.patch
>
>
> It would be useful for analysis purposes to enhance the HLog pretty printer 
> to optionally print a bunch of additional stats such as:
> 1) # of txns
> 2) # of KVs updated
> 3) avg size of txns
> 4) avg size of KVs
> 5) avg # of KVs written per txn
> 5) unique CF signatures involved in put/delete operatons; and breakdown of 
> some of the above metrics by these signatures, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265462#comment-13265462
 ] 

Hadoop QA commented on HBASE-5897:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12525125/testRegionServerCoprocessorExceptionWithRemove.stack
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 6 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1697//console

This message is automatically generated.

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, hbase-5897.txt, 
> testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5860) splitlogmanager should not unnecessarily resubmit tasks when zk unavailable

[Prakash Khemani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265461#comment-13265461
 ] 

Prakash Khemani commented on HBASE-5860:


I had missed the fact that isAnyCreateZKNodePending() misses the create of 
RESCAN nodes. Will provide a fix.

I was aware of the race condition where isAnyCreateZKNodePending() will return 
false even when create-zknode is soon going to be retried. Not worth fixing for 
the reason you outlined - creating an extra RESCAN node doesn't hurt. (The code 
change you have outlined will need some more changes to make it work)

> splitlogmanager should not unnecessarily resubmit tasks when zk unavailable
> ---
>
> Key: HBASE-5860
> URL: https://issues.apache.org/jira/browse/HBASE-5860
> Project: HBase
>  Issue Type: Improvement
>Reporter: Prakash Khemani
>Assignee: Prakash Khemani
> Attachments: 
> 0001-HBASE-5860-splitlogmanager-should-not-unnecessarily-.patch
>
>
> (Doesn't really impact the run time or correctness of log splitting)
> say the master has lost connection to zk. splitlogmanager's timeoutmanager 
> will realize that all the tasks that were submitted are still unassigned. It 
> will resubmit those tasks (i.e. create dummy znodes)
> splitlogmanager should realze that the tasks are unassigned but their znodes 
> have not been created.
> 012-04-20 13:11:20,516 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> dead splitlog worker msgstore295.snc4.facebook.com,60020,1334948757026
> 2012-04-20 13:11:20,517 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> Scheduling batch of logs to split
> 2012-04-20 13:11:20,517 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> started splitting logs in 
> [hdfs://msgstore215.snc4.facebook.com:9000/MSGSTORE215-SNC4-HBASE/.logs/msgstore295.snc4.facebook.com,60020,1334948757026-splitting]
> 2012-04-20 13:11:20,565 INFO org.apache.zookeeper.ClientCnxn: Opening socket 
> connection to server msgstore235.snc4.facebook.com/10.30.222.186:2181
> 2012-04-20 13:11:20,566 INFO org.apache.zookeeper.ClientCnxn: Socket 
> connection established to msgstore235.snc4.facebook.com/10.30.222.186:2181, 
> initiating session
> 2012-04-20 13:11:20,575 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
> total tasks = 4 unassigned = 4
> 2012-04-20 13:11:20,576 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,577 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: 
> resubmitting unassigned task(s) after timeout
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x36ccb0f8010002, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,683 INFO org.apache.zookeeper.ClientCnxn: Unable to read 
> additional data from server sessionid 0x136ccb0f489, likely server has 
> closed socket, closing socket connection and attempting reconnect
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951586677
>  retry=3
> 2012-04-20 13:11:21,786 WARN 
> org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback: create rc 
> =CONNECTIONLOSS for 
> /hbase/splitlog/hdfs%3A%2F%2Fmsgstore215.snc4.facebook.com%3A9000%2FMSGSTORE215-SNC4-HBASE%2F.logs%2Fmsgstore295.snc4.facebook.com%2C60020%2C1334948757026-splitting%2F10.30.251.186%253A60020.1334951920332
>  retry=3

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5879) Enable JMX metrics collection for the Thrift proxy

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5879:
-

Fix Version/s: (was: 0.89-fb)

Removing 0.89fb as fix since it doesn't seem like its been applied there yet, 
only to apache trunk.

> Enable JMX metrics collection for the Thrift proxy
> --
>
> Key: HBASE-5879
> URL: https://issues.apache.org/jira/browse/HBASE-5879
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5879_trunk.txt, D2955.1.patch
>
>
> We need to enable JMX on the Thrift proxy on a separate port different from 
> the JMX port used by regionserver. This is necessary for metrics collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5879) Enable JMX metrics collection for the Thrift proxy

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5879:
-

Fix Version/s: 0.89-fb
   0.96.0

> Enable JMX metrics collection for the Thrift proxy
> --
>
> Key: HBASE-5879
> URL: https://issues.apache.org/jira/browse/HBASE-5879
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Priority: Minor
> Fix For: 0.96.0, 0.89-fb
>
> Attachments: 5879_trunk.txt, D2955.1.patch
>
>
> We need to enable JMX on the Thrift proxy on a separate port different from 
> the JMX port used by regionserver. This is necessary for metrics collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5897:
--

Attachment: testRegionServerCoprocessorExceptionWithRemove.stack

Running the patch against hadoop 1.0 and 0.22, I got 
TestRegionServerCoprocessorExceptionWithRemove hanging on MacBook.

jstack attached.

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, hbase-5897.txt, 
> testRegionServerCoprocessorExceptionWithRemove.stack
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5879) Enable JMX metrics collection for the Thrift proxy

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5879:
-

Attachment: 5879_trunk.txt

What I applied to trunk.

> Enable JMX metrics collection for the Thrift proxy
> --
>
> Key: HBASE-5879
> URL: https://issues.apache.org/jira/browse/HBASE-5879
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Priority: Minor
> Fix For: 0.96.0, 0.89-fb
>
> Attachments: 5879_trunk.txt, D2955.1.patch
>
>
> We need to enable JMX on the Thrift proxy on a separate port different from 
> the JMX port used by regionserver. This is necessary for metrics collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5385) Delete table/column should delete stored permissions on -acl- table

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265451#comment-13265451
 ] 

Matteo Bertozzi commented on HBASE-5385:


{quote}
On preCreateTable and preAddColumn, ensure that the acl table is empty for the 
table / column. We might still have residual acl entries if smt goes wrong. If 
so, we should refuse creating a table by throwing a kind of access control 
exception.
{quote}

Currently there's no check on grant to see if the table/family/qualifier exist. 
Maybe we can open another jira for this, to implement the exists check on grant 
and verify in all pre* if there's nothing left.

> Delete table/column should delete stored permissions on -acl- table  
> -
>
> Key: HBASE-5385
> URL: https://issues.apache.org/jira/browse/HBASE-5385
> Project: HBase
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.94.0
>Reporter: Enis Soztutar
>Assignee: Matteo Bertozzi
> Attachments: HBASE-5385-v0.patch, HBASE-5385-v1.patch
>
>
> Deleting the table or a column does not cascade to the stored permissions at 
> the -acl- table. We should also remove those permissions, otherwise, it can 
> be a security leak, where freshly created tables contain permissions from 
> previous same-named tables. We might also want to ensure, upon table 
> creation, that no entries are already stored at the -acl- table. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265445#comment-13265445
 ] 

Lars Hofhansl commented on HBASE-5897:
--

Yep. I was just wondering whether this is a rather artificial scenario. Anyway, 
your patch makes it much better and actually ensures correct behavior.

Let's commit, unless there're objections.


> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, hbase-5897.txt
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265440#comment-13265440
 ] 

Hadoop QA commented on HBASE-5869:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12525109/5869v9.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 47 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.client.TestScannerTimeout
  org.apache.hadoop.hbase.client.TestMultiParallel
  org.apache.hadoop.hbase.master.TestMasterFailover
  org.apache.hadoop.hbase.TestFullLogReconstruction
  
org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing
  org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
  org.apache.hadoop.hbase.TestDrainingServer
  org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
  org.apache.hadoop.hbase.master.TestDistributedLogSplitting

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1694//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1694//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1694//console

This message is automatically generated.

> Move SplitLogManager splitlog taskstate and AssignmentManager 
> RegionTransitionData znode datas to pb 
> -
>
> Key: HBASE-5869
> URL: https://issues.apache.org/jira/browse/HBASE-5869
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, 
> secondcut.txt, v4.txt, v5.txt, v6.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog

[Gregory Chanan (JIRA)]

Gregory Chanan created HBASE-5908:
-

 Summary: 
TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use 
append to corrupt the HLog
 Key: HBASE-5908
 URL: https://issues.apache.org/jira/browse/HBASE-5908
 Project: HBase
  Issue Type: Bug
  Components: test, wal
Affects Versions: 0.96.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor


TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against a 
version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230

The failure:
"java.io.IOException: Append is not supported. Please see the 
dfs.support.append configuration parameter."

Instead of using append, we can probably just:
- copy over the contents to a new file
- append the garbage to the new file
- copy back to the old file

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5842) Passing shell commands as an argument

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265441#comment-13265441
 ] 

stack commented on HBASE-5842:
--

@Jon http://svn.apache.org/viewvc/hbase/tags/0.18.1/bin/ has a Formatter.rb 
class.  You have to go back further to find our being able to format output as 
html: 
http://svn.apache.org/viewvc/hbase/tags/0.1.3/src/java/org/apache/hadoop/hbase/hql/TableFormatterFactory.java?view=markup

@Harsh -c sounds good easier than echo blah blah.

> Passing shell commands as an argument
> -
>
> Key: HBASE-5842
> URL: https://issues.apache.org/jira/browse/HBASE-5842
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Affects Versions: 0.94.0
>Reporter: Harsh J
>Priority: Minor
>
> Many times we've required scans of .META. to analyze issues with the cluster 
> we work on, and to have the result in a file we can pass around we usually 
> end up doing something like:
> {{echo "scan '.META.'| hbase shell > meta-scan.txt}}
> This can rather be simplified as something like the following instead, with 
> support for a commands reading argument:
> {{hbase shell -c "scan '.META.'"}}
> [Note though: File reading is possible already, i.e. {{hbase shell file.hs}}, 
> but then thats two steps and we usually don't keep a file around for just a 
> meta table scan.]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5879) Enable JMX metrics collection for the Thrift proxy

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265439#comment-13265439
 ] 

Phabricator commented on HBASE-5879:


mbautin has commented on the revision "[jira] [HBASE-5879] [89-fb] Enable JMX 
metrics collection for the Thrift proxy".

  Stack: please do, if you have time! Otherwise, I will port it later. Thanks!

REVISION DETAIL
  https://reviews.facebook.net/D2955

BRANCH
  enable_jmx_metrics_collection_for_the_thrift_HBASE-5879


> Enable JMX metrics collection for the Thrift proxy
> --
>
> Key: HBASE-5879
> URL: https://issues.apache.org/jira/browse/HBASE-5879
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Priority: Minor
> Attachments: D2955.1.patch
>
>
> We need to enable JMX on the Thrift proxy on a separate port different from 
> the JMX port used by regionserver. This is necessary for metrics collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5907) enhance HLog pretty printer to print additional useful stats

[Kannan Muthukkaruppan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265438#comment-13265438
 ] 

Kannan Muthukkaruppan commented on HBASE-5907:
--

working on a patch for this.

> enhance HLog pretty printer to print additional useful stats
> 
>
> Key: HBASE-5907
> URL: https://issues.apache.org/jira/browse/HBASE-5907
> Project: HBase
>  Issue Type: Improvement
>Reporter: Kannan Muthukkaruppan
>Priority: Minor
>
> It would be useful for analysis purposes to enhance the HLog pretty printer 
> to optionally print a bunch of additional stats such as:
> 1) # of txns
> 2) # of KVs updated
> 3) avg size of txns
> 4) avg size of KVs
> 5) avg # of KVs written per txn
> 5) unique CF signatures involved in put/delete operatons; and breakdown of 
> some of the above metrics by these signatures, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-5884) MapReduce package info has broken link to bulk-loads

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-5884.
--

   Resolution: Fixed
Fix Version/s: (was: 0.94.1)
   0.94.0
 Hadoop Flags: Reviewed

Applied 0.94 and trunk.  Thanks for the patch Jesse.

> MapReduce package info has broken link to bulk-loads
> 
>
> Key: HBASE-5884
> URL: https://issues.apache.org/jira/browse/HBASE-5884
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.96.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
>Priority: Trivial
> Fix For: 0.94.0, 0.96.0
>
> Attachments: doc_HBASE-5884.patch
>
>
> Bulk Loads link goes to an old link, which we have dropped recently.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5907) enhance HLog pretty printer to print additional useful stats

[Kannan Muthukkaruppan (JIRA)]

Kannan Muthukkaruppan created HBASE-5907:


 Summary: enhance HLog pretty printer to print additional useful 
stats
 Key: HBASE-5907
 URL: https://issues.apache.org/jira/browse/HBASE-5907
 Project: HBase
  Issue Type: Improvement
Reporter: Kannan Muthukkaruppan
Priority: Minor


It would be useful for analysis purposes to enhance the HLog pretty printer to 
optionally print a bunch of additional stats such as:

1) # of txns
2) # of KVs updated
3) avg size of txns
4) avg size of KVs
5) avg # of KVs written per txn
5) unique CF signatures involved in put/delete operatons; and breakdown of some 
of the above metrics by these signatures, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5879) Enable JMX metrics collection for the Thrift proxy

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265436#comment-13265436
 ] 

Phabricator commented on HBASE-5879:


stack has commented on the revision "[jira] [HBASE-5879] [89-fb] Enable JMX 
metrics collection for the Thrift proxy".

  Mikhail, I should just apply this to trunk?   It looks like it'd go in easy.

REVISION DETAIL
  https://reviews.facebook.net/D2955

BRANCH
  enable_jmx_metrics_collection_for_the_thrift_HBASE-5879


> Enable JMX metrics collection for the Thrift proxy
> --
>
> Key: HBASE-5879
> URL: https://issues.apache.org/jira/browse/HBASE-5879
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Priority: Minor
> Attachments: D2955.1.patch
>
>
> We need to enable JMX on the Thrift proxy on a separate port different from 
> the JMX port used by regionserver. This is necessary for metrics collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer

[Li Pi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265426#comment-13265426
 ] 

Li Pi commented on HBASE-5699:
--

Agree with todd on the implementation details. The switching of logs should 
also serve to help balance our log writes. 

> Run with > 1 WAL in HRegionServer
> -
>
> Key: HBASE-5699
> URL: https://issues.apache.org/jira/browse/HBASE-5699
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Li Pi
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265425#comment-13265425
 ] 

Todd Lipcon commented on HBASE-5886:


To fill in some color, we had a customer complaining of data loss after a 
crash. We spent many hours doing post mortem analysis, and only after we ruled 
out lots and lots of corner case possibilities did we come to find out that the 
user's application had set the writeToWAL flag false in a small number of their 
puts. Having the log message and the metric make it easy to say "yea, you got 
data loss, but here's a smoking gun which shows you explicitly disabled WAL for 
some puts in that region". This is important in lots of environments where 
HBase ops/support may be distinct from the users writing applications, and may 
not even have access to the source code of those applications.

> Add new metric for possible data loss due to puts without WAL 
> --
>
> Key: HBASE-5886
> URL: https://issues.apache.org/jira/browse/HBASE-5886
> Project: HBase
>  Issue Type: New Feature
>  Components: metrics, regionserver
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
>  Labels: metrics
> Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch, 
> HBASE-5886-v2.patch
>
>
> Add a metrics to keep track of puts without WAL and possible data loss size.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265421#comment-13265421
 ] 

Todd Lipcon commented on HBASE-5699:


bq. I disagree, considering that most of the deployments have rep=3 you're 
using three spindles not one

That said, most of our customers are deploying 6 disks if not 12 :)

IMO the other big gain we can get from multiple WALs is to automatically switch 
between WALs when one gets "slow". IMO we should maintain a count of 
outstanding requests (probably by size) for each WAL, and submit writes to 
whichever has fewer outstanding requests. That way if one is faster, it will 
take more of the load. Then simultaneously measure trailing latency stats on 
each WAL, and if one is significantly slower than the other for some period of 
time, have it roll (to try to get a new set of disks/nodes)

> Run with > 1 WAL in HRegionServer
> -
>
> Key: HBASE-5699
> URL: https://issues.apache.org/jira/browse/HBASE-5699
> Project: HBase
>  Issue Type: Improvement
>Reporter: binlijin
>Assignee: Li Pi
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265414#comment-13265414
 ] 

Todd Lipcon commented on HBASE-5897:


bq. So calling doMiniBatchPut multiple times on behalf of the same Put 
operation should be happening relatively rarely.

True, unless there's a lot of contention on some rows. In the test I was doing, 
I was slamming a fairly small number of rows with puts, so there was a lot of 
this "back-off" behavior.

I bet we could improve performance by making it continue past a failed lock 
acquisition and acquire as many as possible, while pushing those it missed back 
onto the end of the list. Right now, as soon as it fails any lock acquisition, 
it stops the batch there (so we always do contiguous ranges of ops instead of 
arbitrary sets)

> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, hbase-5897.txt
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5888) Clover profile in build

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-5888:
-

Status: Patch Available  (was: Open)

> Clover profile in build
> ---
>
> Key: HBASE-5888
> URL: https://issues.apache.org/jira/browse/HBASE-5888
> Project: HBase
>  Issue Type: Improvement
>  Components: build, test
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-clover_v1.patch, hbase-clover_v2.patch
>
>
> Clover is disabled right now. I would like to add a profile that enables 
> clover reports. We can also backport this to 0.92, and 0.94, since we are 
> also interested in test coverage for those branches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5888) Clover profile in build

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-5888:
-

Attachment: hbase-clover_v2.patch

Uploaded wrong patch. This should be the one.

> Clover profile in build
> ---
>
> Key: HBASE-5888
> URL: https://issues.apache.org/jira/browse/HBASE-5888
> Project: HBase
>  Issue Type: Improvement
>  Components: build, test
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-clover_v1.patch, hbase-clover_v2.patch
>
>
> Clover is disabled right now. I would like to add a profile that enables 
> clover reports. We can also backport this to 0.92, and 0.94, since we are 
> also interested in test coverage for those branches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5888) Clover profile in build

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-5888:
-

Attachment: (was: HBASE-5358_v2.patch)

> Clover profile in build
> ---
>
> Key: HBASE-5888
> URL: https://issues.apache.org/jira/browse/HBASE-5888
> Project: HBase
>  Issue Type: Improvement
>  Components: build, test
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-clover_v1.patch
>
>
> Clover is disabled right now. I would like to add a profile that enables 
> clover reports. We can also backport this to 0.92, and 0.94, since we are 
> also interested in test coverage for those branches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5888) Clover profile in build

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-5888:
-

Status: Open  (was: Patch Available)

> Clover profile in build
> ---
>
> Key: HBASE-5888
> URL: https://issues.apache.org/jira/browse/HBASE-5888
> Project: HBase
>  Issue Type: Improvement
>  Components: build, test
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-clover_v1.patch
>
>
> Clover is disabled right now. I would like to add a profile that enables 
> clover reports. We can also backport this to 0.92, and 0.94, since we are 
> also interested in test coverage for those branches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-5548) Add ability to get a table in the shell

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-5548.
--

   Resolution: Fixed
Fix Version/s: (was: 0.94.1)
 Hadoop Flags: Reviewed

Committed to trunk after poking around with it for a while. It looks good. 
Thanks for the patch Jesse.

> Add ability to get a table in the shell
> ---
>
> Key: HBASE-5548
> URL: https://issues.apache.org/jira/browse/HBASE-5548
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.96.0
>
> Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, 
> ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch
>
>
> Currently, all the commands that operate on a table in the shell first have 
> to take the table as name as input. 
> There are two main considerations:
> * It is annoying to have to write the table name every time, when you should 
> just be able to get a reference to a table
> * the current implementation is very wasteful - it creates a new HTable for 
> each call (but reuses the connection since it uses the same configuration)
> We should be able to get a handle to a single HTable and then operate on that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5888) Clover profile in build

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265404#comment-13265404
 ] 

Hadoop QA commented on HBASE-5888:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12525115/HBASE-5358_v2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 11 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1695//console

This message is automatically generated.

> Clover profile in build
> ---
>
> Key: HBASE-5888
> URL: https://issues.apache.org/jira/browse/HBASE-5888
> Project: HBase
>  Issue Type: Improvement
>  Components: build, test
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-5358_v2.patch, hbase-clover_v1.patch
>
>
> Clover is disabled right now. I would like to add a profile that enables 
> clover reports. We can also backport this to 0.92, and 0.94, since we are 
> also interested in test coverage for those branches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5385) Delete table/column should delete stored permissions on -acl- table

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265400#comment-13265400
 ] 

Hadoop QA commented on HBASE-5385:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12525101/HBASE-5385-v1.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1693//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1693//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1693//console

This message is automatically generated.

> Delete table/column should delete stored permissions on -acl- table  
> -
>
> Key: HBASE-5385
> URL: https://issues.apache.org/jira/browse/HBASE-5385
> Project: HBase
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.94.0
>Reporter: Enis Soztutar
>Assignee: Matteo Bertozzi
> Attachments: HBASE-5385-v0.patch, HBASE-5385-v1.patch
>
>
> Deleting the table or a column does not cascade to the stored permissions at 
> the -acl- table. We should also remove those permissions, otherwise, it can 
> be a security leak, where freshly created tables contain permissions from 
> previous same-named tables. We might also want to ensure, upon table 
> creation, that no entries are already stored at the -acl- table. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265399#comment-13265399
 ] 

Lars Hofhansl commented on HBASE-5897:
--

Looking at the code some more... doMiniBactchPut will handle the entire batch 
unless it cannot lock the involved rows (see STEP 1. in doMiniBatchPut).
So calling doMiniBatchPut multiple times on behalf of the same Put operation 
should be happening relatively rarely.


> prePut coprocessor hook causing substantial CPU usage
> -
>
> Key: HBASE-5897
> URL: https://issues.apache.org/jira/browse/HBASE-5897
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5897-simple.txt, hbase-5897.txt
>
>
> I was running an insert workload against trunk under oprofile and saw that a 
> significant portion of CPU usage was going to calling the "prePut" 
> coprocessor hook inside doMiniBatchPut, even though I don't have any 
> coprocessors installed. I ran a million-row insert and collected CPU time 
> spent in the RS after commenting out the preput hook, and found CPU usage 
> reduced by 33%.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5888) Clover profile in build

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-5888:
-

Status: Patch Available  (was: Open)

> Clover profile in build
> ---
>
> Key: HBASE-5888
> URL: https://issues.apache.org/jira/browse/HBASE-5888
> Project: HBase
>  Issue Type: Improvement
>  Components: build, test
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-5358_v2.patch, hbase-clover_v1.patch
>
>
> Clover is disabled right now. I would like to add a profile that enables 
> clover reports. We can also backport this to 0.92, and 0.94, since we are 
> also interested in test coverage for those branches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5888) Clover profile in build

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-5888:
-

Attachment: HBASE-5358_v2.patch

Updated the patch to ignore generated packages (thrift.generated, 
protobuf.generated), since they are skewing coverage results. 

I uploaded a sample report for 0.92 here:
http://people.apache.org/~enis/hbase-clover/

> Clover profile in build
> ---
>
> Key: HBASE-5888
> URL: https://issues.apache.org/jira/browse/HBASE-5888
> Project: HBase
>  Issue Type: Improvement
>  Components: build, test
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-5358_v2.patch, hbase-clover_v1.patch
>
>
> Clover is disabled right now. I would like to add a profile that enables 
> clover reports. We can also backport this to 0.92, and 0.94, since we are 
> also interested in test coverage for those branches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5890) SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5890:
-

Fix Version/s: (was: 0.94.0)
   0.94.1

Moving out for now.

> SplitLog Rescan BusyWaits upon Zk.CONNECTIONLOSS
> 
>
> Key: HBASE-5890
> URL: https://issues.apache.org/jira/browse/HBASE-5890
> Project: HBase
>  Issue Type: Bug
>Reporter: Nicolas Spiegelberg
>Priority: Minor
> Fix For: 0.96.0, 0.89-fb, 0.94.1
>
> Attachments: HBASE-5890.patch
>
>
> We ran into a production issue yesterday where the SplitLogManager tried to 
> create a Rescan node in ZK.  The createAsync() generated a 
> KeeperException.CONNECTIONLOSS that was immedately sent to processResult(), 
> createRescan node with --retry_count was called, and this created a CPU 
> busywait that also clogged up the logs.  We should handle this better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5906) TestChangingEncoding failing sporadically in 0.94 build

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265384#comment-13265384
 ] 

Hudson commented on HBASE-5906:
---

Integrated in HBase-0.94 #161 (See 
[https://builds.apache.org/job/HBase-0.94/161/])
HBASE-5906 TestChangingEncoding failing sporadically in 0.94 build 
(Revision 1332319)

 Result = FAILURE
stack : 
Files : 
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java


> TestChangingEncoding failing sporadically in 0.94 build
> ---
>
> Key: HBASE-5906
> URL: https://issues.apache.org/jira/browse/HBASE-5906
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
> Attachments: 5906.txt
>
>
> The test passes locally for me and Elliott but takes a long time to run.  
> Timeout is only two minutes for the test though.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265383#comment-13265383
 ] 

Hudson commented on HBASE-5611:
---

Integrated in HBase-0.94 #161 (See 
[https://builds.apache.org/job/HBase-0.94/161/])
HBASE-5611 Replayed edits from regions that failed to open during recovery 
aren't removed from the global MemStore size - v2 (Jieshan) (Revision 1332344)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerAccounting.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java


> Replayed edits from regions that failed to open during recovery aren't 
> removed from the global MemStore size
> 
>
> Key: HBASE-5611
> URL: https://issues.apache.org/jira/browse/HBASE-5611
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.6
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Critical
> Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
> Attachments: 5611-94-v2.txt, 5611-94.addendum, HBASE-5611-92.patch, 
> HBASE-5611-94-minorchange.patch, HBASE-5611-trunk-v2-minorchange.patch
>
>
> This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think 
> it's still possible to hit it if a region fails to open for more obscure 
> reasons like HDFS errors.
> Consider a region that just went through distributed splitting and that's now 
> being opened by a new RS. The first thing it does is to read the recovery 
> files and put the edits in the {{MemStores}}. If this process takes a long 
> time, the master will move that region away. At that point the edits are 
> still accounted for in the global {{MemStore}} size but they are dropped when 
> the {{HRegion}} gets cleaned up. It's completely invisible until the 
> {{MemStoreFlusher}} needs to force flush a region and that none of them have 
> edits:
> {noformat}
> 2012-03-21 00:33:39,303 DEBUG 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up 
> because memory above low water=5.9g
> 2012-03-21 00:33:39,303 ERROR 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed 
> for entry null
> java.lang.IllegalStateException
> at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> The {{null}} here is a region. In my case I had so many edits in the 
> {{MemStore}} during recovery that I'm over the low barrier although in fact 
> I'm at 0. It happened yesterday and it still printing this out.
> To fix this we need to be able to decrease the global {{MemStore}} size when 
> the region can't open.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5869:
-

Attachment: 5869v9.txt

I was returning early in AssignmentManager if null data inside isCarryingRegion 
when I should have carried on to trip over the get of region location from the 
AM memory.  Seems to fix some of the failing tests.

> Move SplitLogManager splitlog taskstate and AssignmentManager 
> RegionTransitionData znode datas to pb 
> -
>
> Key: HBASE-5869
> URL: https://issues.apache.org/jira/browse/HBASE-5869
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, 
> secondcut.txt, v4.txt, v5.txt, v6.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira