date:20111028


 [ 
https://issues.apache.org/jira/browse/HBASE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4304:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Applied branch and trunk.  Thanks for the patch Li.

 requestsPerSecond counter stuck at 0
 

 Key: HBASE-4304
 URL: https://issues.apache.org/jira/browse/HBASE-4304
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Li Pi
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4304v1.txt, 4304v2.txt, 4304v3.txt, 4304v4.txt


 Running trunk @ r1163343, all of the requestsPerSecond counters are showing 0 
 both in the master UI and in the RS UI. The writeRequestsCount metric is 
 properly updating in the RS UI.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4304) requestsPerSecond counter stuck at 0


 [ 
https://issues.apache.org/jira/browse/HBASE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4304:
-

Attachment: 4304v4.txt

Here is what I applied.  I tested it too.  Seems to work.

 requestsPerSecond counter stuck at 0
 

 Key: HBASE-4304
 URL: https://issues.apache.org/jira/browse/HBASE-4304
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Li Pi
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4304v1.txt, 4304v2.txt, 4304v3.txt, 4304v4.txt


 Running trunk @ r1163343, all of the requestsPerSecond counters are showing 0 
 both in the master UI and in the RS UI. The writeRequestsCount metric is 
 properly updating in the RS UI.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4691) Remove more unnecessary byte[] copies from KeyValues


[ 
https://issues.apache.org/jira/browse/HBASE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138105#comment-13138105
 ] 

Ted Yu commented on HBASE-4691:
---

Patch looks fine to me.

 Remove more unnecessary byte[] copies from KeyValues
 

 Key: HBASE-4691
 URL: https://issues.apache.org/jira/browse/HBASE-4691
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.94.0

 Attachments: 4691.txt


 Just looking through the code I found some more spots where we unnecessarily 
 copy byte[] rather than just passing offset and length around.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4681) StringIndexOutOfBoundsException parsing Hostname


[ 
https://issues.apache.org/jira/browse/HBASE-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138107#comment-13138107
 ] 

stack commented on HBASE-4681:
--

This should be fixed after HBASE-4692 goes in.

 StringIndexOutOfBoundsException parsing Hostname
 

 Key: HBASE-4681
 URL: https://issues.apache.org/jira/browse/HBASE-4681
 Project: HBase
  Issue Type: Bug
Reporter: stack
 Fix For: 0.92.0


 Starting a 0.92 on 0.90 data with 0.90 zk ensemble I got this:
 {code}
 2011-10-26 06:13:53,920 ERROR 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master 
 already exists and this is not a retry
 2011-10-26 06:13:53,927 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.StringIndexOutOfBoundsException: String index out of range: -1
 at java.lang.String.substring(String.java:1937)
 at 
 org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81)
 at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63)
 at 
 org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148)
 at 
 org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:346)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301)
 at java.lang.Thread.run(Thread.java:662)
 2011-10-26 06:13:53,929 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
 2011-10-26 06:13:53,929 DEBUG org.apache.hadoop.hbase.master.HMaster: 
 Stopping service thre
 {code}
 I thought this had been fixed.  Dig in .

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4681) StringIndexOutOfBoundsException parsing Hostname


[ 
https://issues.apache.org/jira/browse/HBASE-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138109#comment-13138109
 ] 

stack commented on HBASE-4681:
--

In current trunk, we were writing the master address without its version 
prefix.  Without it we were not able to read the master znode content.

 StringIndexOutOfBoundsException parsing Hostname
 

 Key: HBASE-4681
 URL: https://issues.apache.org/jira/browse/HBASE-4681
 Project: HBase
  Issue Type: Bug
Reporter: stack
 Fix For: 0.92.0


 Starting a 0.92 on 0.90 data with 0.90 zk ensemble I got this:
 {code}
 2011-10-26 06:13:53,920 ERROR 
 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master 
 already exists and this is not a retry
 2011-10-26 06:13:53,927 FATAL org.apache.hadoop.hbase.master.HMaster: 
 Unhandled exception. Starting shutdown.
 java.lang.StringIndexOutOfBoundsException: String index out of range: -1
 at java.lang.String.substring(String.java:1937)
 at 
 org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81)
 at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63)
 at 
 org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148)
 at 
 org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:346)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301)
 at java.lang.Thread.run(Thread.java:662)
 2011-10-26 06:13:53,929 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
 2011-10-26 06:13:53,929 DEBUG org.apache.hadoop.hbase.master.HMaster: 
 Stopping service thre
 {code}
 I thought this had been fixed.  Dig in .

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4641) Block cache can be mistakenly instantiated on Master


[ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138114#comment-13138114
 ] 

Ted Yu commented on HBASE-4641:
---

Initially I was thinking of resorting to:
{code}
StackTraceElement[] stackTraceElements =
  Thread.currentThread().getStackTrace();
{code}
The above is ugly too.

Can HMaster add one more entry to its Configuration conf which signifies that 
the conf is not owned by region server ?
Store can check this entry and make decision accordingly.

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, HBASE-4641-v1.patch, 
 HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4691) Remove more unnecessary byte[] copies from KeyValues


[ 
https://issues.apache.org/jira/browse/HBASE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138115#comment-13138115
 ] 

stack commented on HBASE-4691:
--

+1

There were a few howlers in there.  E.g:

{code}
-if (kv == null || kv.getValue().length != Bytes.SIZEOF_LONG)
+if (kv == null || kv.getValueLength() != Bytes.SIZEOF_LONG)
{code}

Create a byte array just to get the length.

And this one:

{code}
   byte[] lastKey = kvs.get(0).getRow();
-  Put put = new Put(kvs.get(0).getRow(),
-  kvs.get(0).getTimestamp());
{code}

Good stuff Lars.

 Remove more unnecessary byte[] copies from KeyValues
 

 Key: HBASE-4691
 URL: https://issues.apache.org/jira/browse/HBASE-4691
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.94.0

 Attachments: 4691.txt


 Just looking through the code I found some more spots where we unnecessarily 
 copy byte[] rather than just passing offset and length around.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4641) Block cache can be mistakenly instantiated on Master


[ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138118#comment-13138118
 ] 

stack commented on HBASE-4641:
--

I'm not sure that will work since the Configuration is shared among the servers 
in the minicluster.

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, HBASE-4641-v1.patch, 
 HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4692) HBASE-4300 broke the build


 [ 
https://issues.apache.org/jira/browse/HBASE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4692:
-

Attachment: 4692-v2.txt

Patch build has distributed log splitting failing but it passes for me locally. 
 Similar for TestSplitTransactionOnCluster.  The TestLogRolling fail I'll dig 
into in morning.  Will commit this patch against this issue so the build gets a 
bit more sane than it currently is.

 HBASE-4300 broke the build
 --

 Key: HBASE-4692
 URL: https://issues.apache.org/jira/browse/HBASE-4692
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4692-v2.txt, 4692.txt


 Rebasing my patch I missed encoding of master name up in zk.  Creating a 
 separate issue so can try patch build with my fix rather than run all tests 
 locally.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4692) HBASE-4300 broke the build

2011-10-28 Thread Hadoop QA (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138121#comment-13138121
]

Hadoop QA commented on HBASE-4692:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501227/4692.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated -167 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.master.TestDistributedLogSplitting

org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.master.TestActiveMasterManager

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/82//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/82//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/82//console

This message is automatically generated.

HBASE-4300 broke the build
--

Key: HBASE-4692
URL: https://issues.apache.org/jira/browse/HBASE-4692
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
Fix For: 0.92.0

Attachments: 4692-v2.txt, 4692.txt

Rebasing my patch I missed encoding of master name up in zk. Creating a
separate issue so can try patch build with my fix rather than run all tests
locally.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4641) Block cache can be mistakenly instantiated on Master


[ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138126#comment-13138126
 ] 

Ted Yu commented on HBASE-4641:
---

I think the fact that Configuration object may be shared by both master and 
region server(s) in minicluster poses limitation on this and future tasks.
We should think of changing this sharing which would never happen in production 
cluster.

Once that is done, there would be no need to replicate Configuration as patch 
v1 did.

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, HBASE-4641-v1.patch, 
 HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4692) HBASE-4300 broke the build


[ 
https://issues.apache.org/jira/browse/HBASE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138125#comment-13138125
 ] 

stack commented on HBASE-4692:
--

The uploaded v2 fixes TestActiveMasterManager fail.  The TestMasterObserver is 
unrelated. Will look at these fails in separate tickets in morning.

 HBASE-4300 broke the build
 --

 Key: HBASE-4692
 URL: https://issues.apache.org/jira/browse/HBASE-4692
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4692-v2.txt, 4692.txt


 Rebasing my patch I missed encoding of master name up in zk.  Creating a 
 separate issue so can try patch build with my fix rather than run all tests 
 locally.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master


[ 
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138129#comment-13138129
 ] 

stack commented on HBASE-4470:
--

You'd have to do it in many places -- I don't think it much use fixing just 
this one locale -- and you'd pull in Callable and steal the bits from 
HConnectionManager to run the retrying.  It'd be a big messy patch.

 ServerNotRunningException coming out of assignRootAndMeta kills the Master
 --

 Key: HBASE-4470
 URL: https://issues.apache.org/jira/browse/HBASE-4470
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Priority: Blocker
 Fix For: 0.90.5


 I'm surprised we still have issues like that and I didn't get a hit while 
 googling so forgive me if there's already a jira about it.
 When the master starts it verifies the locations of root and meta before 
 assigning them, if the server is started but not running you'll get this:
 {quote}
 2011-09-23 04:47:44,859 WARN 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
 RemoteException connecting to RS
 org.apache.hadoop.ipc.RemoteException: 
 org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running 
 yet
 at 
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
 at 
 org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
 at $Proxy6.getProtocolVersion(Unknown Source)
 at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
 at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
 at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
 at 
 org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
 at 
 org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
 at 
 org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
 at 
 org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
 at 
 org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
 at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
 {quote}
 I hit that 3-4 times this week while debugging something else. The worst is 
 that when you restart the master it sees that as a failover, but none of the 
 regions are assigned so it takes an eternity to get back fully online.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4692) HBASE-4300 broke the build

2011-10-28 Thread Hadoop QA (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138134#comment-13138134
 ] 

Hadoop QA commented on HBASE-4692:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12501246/4692-v2.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 6 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/84//console

This message is automatically generated.

 HBASE-4300 broke the build
 --

 Key: HBASE-4692
 URL: https://issues.apache.org/jira/browse/HBASE-4692
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4692-v2.txt, 4692.txt


 Rebasing my patch I missed encoding of master name up in zk.  Creating a 
 separate issue so can try patch build with my fix rather than run all tests 
 locally.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4679) Thrift null mutation error

2011-10-28 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138172#comment-13138172
 ] 

Hudson commented on HBASE-4679:
---

Integrated in HBase-TRUNK #2377 (See 
[https://builds.apache.org/job/HBase-TRUNK/2377/])
HBASE-4679 Thrift null mutation error

nspiegelberg : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java


 Thrift null mutation error
 --

 Key: HBASE-4679
 URL: https://issues.apache.org/jira/browse/HBASE-4679
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
 Fix For: 0.92.0, 0.94.0

 Attachments: HBASE-4679.patch


 When using null as a value for a mutation, HBasse thrift client failed and 
 threw an error. We should instad check for a null byte buffer.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4577) Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB

2011-10-28 Thread gaojinchao (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138182#comment-13138182
 ] 

gaojinchao commented on HBASE-4577:
---

It seems result is ok.
numberOfStores=1, numberOfStorefiles=12, storefileUncompressedSizeMB=2344, 
storefileSizeMB=2344, compressionRatio=1., memstoreSizeMB=100, 
storefileIndexSizeMB=1, readRequestsCount=0, writeRequestsCount=35004, 
rootIndexSizeKB=1045, totalStaticIndexSizeKB=1935, totalStaticBloomSizeKB=0, 
totalCompactingKVs=5481732, currentCompactedKVs=5132546, 
compactionProgressPct=0.0, coprocessors=[]

rawDataSize is instead of uncompressedBytes ? all docs use uncompressedBytes.

 Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
 -

 Key: HBASE-4577
 URL: https://issues.apache.org/jira/browse/HBASE-4577
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBASE-4577_trial_Trunk.patch


 Minor issue while looking at the RS metrics:
 bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, 
 storefileSizeMB=2420, compressionRatio=1.0008
 I guess there's a truncation somewhere when it's adding the numbers up.
 FWIW there's no compression on that table.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4120) isolation and allocation


[ 
https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138196#comment-13138196
 ] 

jirapos...@reviews.apache.org commented on HBASE-4120:
--



bq.  On 2011-10-26 21:54:57, Ted Yu wrote:
bq.   
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java,
 line 131
bq.   https://reviews.apache.org/r/1421/diff/6/?file=53369#file53369line131
bq.  
bq.   Should read 'priority should be between'.
bq.   
bq.   So only region priority can be negative ?

the priority user set must between 1~10,the negative priority is used by some 
system table like meta and root,or other urgency.
If an IPC call is waiting too long, it's priority may increase to a negative 
number.


- Jia


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1421/#review2869
---


On 2011-10-27 13:05:36, Jia Liu wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1421/
bq.  ---
bq.  
bq.  (Updated 2011-10-27 13:05:36)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Patch used for table priority alone,In this patch, not only tables can 
have different priorities but also the different actions like 
get,scan,put and delete can have priorities.
bq.  
bq.  
bq.  This addresses bug HBase-4120.
bq.  https://issues.apache.org/jira/browse/HBase-4120
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
 1189169 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1189169 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1189169 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/1421/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Tested with test cases in  TestCase_For_TablePriority_trunk_v1.patch 
bq.  please apply the patch of HBASE-4181 first,in some circumstances this bug 
will affect the performance of client.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jia
bq.  
bq.



 isolation and allocation
 

 Key: HBASE-4120
 URL: https://issues.apache.org/jira/browse/HBASE-4120
 Project: HBase
  Issue Type: New Feature
  Components: master, regionserver
Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0
Reporter: Liu Jia
 Fix For: 0.94.0

 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, 
 Design_document_for_HBase_isolation_and_allocation_Revised.pdf, 
 HBase_isolation_and_allocation_user_guide.pdf, 
 Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch


 The HBase isolation and allocation tool is designed to help users manage 
 cluster resource among different application and tables.
 When we have a large scale of HBase cluster with many applications running on 
 it, there will be lots of problems. In Taobao there is a cluster for many 
 departments to test their applications performance, these applications are 
 based on HBase. With one cluster which has 12 servers, there will be only one 
 application running exclusively on this server, and many other applications 
 must wait until the previous test finished.
 After we add allocation manage function to the cluster, applications can 
 share the cluster and run concurrently. Also if the Test Engineer wants to 
 make sure there is no interference, he/she can move out other tables from 
 this group.
 In groups we use table priority to allocate resource, when system is busy; we 
 can make sure high-priority tables

[jira] [Commented] (HBASE-4120) isolation and allocation

2011-10-28 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138197#comment-13138197
]

jirapos...@reviews.apache.org commented on HBASE-4120:
--

bq. On 2011-08-09 14:39:49, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java,
line 534
bq. https://reviews.apache.org/r/1421/diff/3/?file=31597#file31597line534
bq.
bq. Return value should be boolean.

this class implements BlockingQueue and put is a override method..so I can't
change the return value.

- Jia

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1421/#review1345
---

On 2011-10-27 13:05:36, Jia Liu wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1421/
bq. ---
bq.
bq. (Updated 2011-10-27 13:05:36)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Patch used for table priority alone,In this patch, not only tables can
have different priorities but also the different actions like
get,scan,put and delete can have priorities.
bq.
bq.
bq. This addresses bug HBase-4120.
bq. https://issues.apache.org/jira/browse/HBase-4120
bq.
bq.
bq. Diffs
bq. -
bq.
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
1189169
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
1189169
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1189169
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1421/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch
bq. please apply the patch of HBASE-4181 first,in some circumstances this bug
will affect the performance of client.
bq.
bq.
bq. Thanks,
bq.
bq. Jia
bq.
bq.

isolation and allocation

Key: HBASE-4120
URL: https://issues.apache.org/jira/browse/HBASE-4120
Project: HBase
Issue Type: New Feature
Components: master, regionserver
Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0
Reporter: Liu Jia
Fix For: 0.94.0

Attachments: Design_document_for_HBase_isolation_and_allocation.pdf,
Design_document_for_HBase_isolation_and_allocation_Revised.pdf,
HBase_isolation_and_allocation_user_guide.pdf,
Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch

The HBase isolation and allocation tool is designed to help users manage
cluster resource among different application and tables.
When we have a large scale of HBase cluster with many applications running on
it, there will be lots of problems. In Taobao there is a cluster for many
departments to test their applications performance, these applications are
based on HBase. With one cluster which has 12 servers, there will be only one
application running exclusively on this server, and many other applications
must wait until the previous test finished.
After we add allocation manage function to the cluster, applications can
share the cluster and run concurrently. Also if the Test Engineer wants to
make sure there is no interference, he/she can move out other tables from
this group.
In groups we use table priority to allocate resource, when system is busy; we
can make sure high-priority tables are not affected lower-priority tables
Different groups can have different region server configurations, some groups
optimized for reading can have large block cache size, and others

[jira] [Commented] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around

2011-10-28 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138241#comment-13138241
 ] 

Hudson commented on HBASE-4300:
---

Integrated in HBase-0.92 #87 (See 
[https://builds.apache.org/job/HBase-0.92/87/])
HBASE-4300 Start of new-version master fails if old master's znode is 
hanging around

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ServerName.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Addressing.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerTracker.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestServerName.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java


 Start of new-version master fails if old master's znode is hanging around
 -

 Key: HBASE-4300
 URL: https://issues.apache.org/jira/browse/HBASE-4300
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: stack
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4300-v2.txt, 4300-v3.txt, 4300.txt


 I shut down an 0.90 cluster, and had to do so uncleanly. I then started a 
 trunk (0.92) cluster before the old master znode had expired. This cased:
 java.lang.StringIndexOutOfBoundsException: String index out of range: -1
 at java.lang.String.substring(String.java:1937)
 at 
 org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81)
 at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63)
 at 
 org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148)
 at 
 org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4120) isolation and allocation


[ 
https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138340#comment-13138340
 ] 

jirapos...@reviews.apache.org commented on HBASE-4120:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1421/
---

(Updated 2011-10-28 13:45:51.987878)


Review request for hbase.


Changes
---

fix bugs of patch-v4


Summary
---

Patch used for table priority alone,In this patch, not only tables can have 
different priorities but also the different actions like get,scan,put and 
delete can have priorities.


This addresses bug HBase-4120.
https://issues.apache.org/jira/browse/HBase-4120


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
 1189169 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1189169 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1189169 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/1421/diff


Testing
---

Tested with test cases in  TestCase_For_TablePriority_trunk_v1.patch 
please apply the patch of HBASE-4181 first,in some circumstances this bug will 
affect the performance of client.


Thanks,

Jia



 isolation and allocation
 

 Key: HBASE-4120
 URL: https://issues.apache.org/jira/browse/HBASE-4120
 Project: HBase
  Issue Type: New Feature
  Components: master, regionserver
Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0
Reporter: Liu Jia
 Fix For: 0.94.0

 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, 
 Design_document_for_HBase_isolation_and_allocation_Revised.pdf, 
 HBase_isolation_and_allocation_user_guide.pdf, 
 Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch


 The HBase isolation and allocation tool is designed to help users manage 
 cluster resource among different application and tables.
 When we have a large scale of HBase cluster with many applications running on 
 it, there will be lots of problems. In Taobao there is a cluster for many 
 departments to test their applications performance, these applications are 
 based on HBase. With one cluster which has 12 servers, there will be only one 
 application running exclusively on this server, and many other applications 
 must wait until the previous test finished.
 After we add allocation manage function to the cluster, applications can 
 share the cluster and run concurrently. Also if the Test Engineer wants to 
 make sure there is no interference, he/she can move out other tables from 
 this group.
 In groups we use table priority to allocate resource, when system is busy; we 
 can make sure high-priority tables are not affected lower-priority tables
 Different groups can have different region server configurations, some groups 
 optimized for reading can have large block cache size, and others optimized 
 for writing can have large memstore size. 
 Tables and region servers can be moved easily between groups; after changing 
 the configuration, a group can be restarted alone instead of restarting the 
 whole cluster.
 git entry : https://github.com/ICT-Ope/HBase_allocation .
 We hope our work is helpful.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4120) isolation and allocation

[
https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138392#comment-13138392
]

Ted Yu commented on HBASE-4120:
---

For metaHandlerCount parameter, we can keep its name since it is for .META. and
ROOT regions.

For now, we can keep PriorityFunction.java in ipc module.

isolation and allocation

[jira] [Commented] (HBASE-2312) Possible data loss when RS goes into GC pause while rolling HLog

2011-10-28 Thread Phabricator (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138426#comment-13138426
 ] 

Phabricator commented on HBASE-2312:


Kannan has added CCs to the revision HBASE-2312 [jira] Possible data loss when 
RS goes into GC pause while rolling HLog.
Added CCs: Kannan

REVISION DETAIL
  https://reviews.facebook.net/D99


 Possible data loss when RS goes into GC pause while rolling HLog
 

 Key: HBASE-2312
 URL: https://issues.apache.org/jira/browse/HBASE-2312
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.90.0
Reporter: Karthik Ranganathan
Assignee: Nicolas Spiegelberg
Priority: Critical
 Fix For: 0.92.0

 Attachments: D99.1.patch


 There is a very corner case when bad things could happen(ie data loss):
 1)RS #1 is going to roll its HLog - not yet created the new one, old one 
 will get no more writes
 2)RS #1 enters GC Pause of Death
 3)Master lists HLog files of RS#1 that is has to split as RS#1 is dead, 
 starts splitting
 4)RS #1 wakes up, created the new HLog (previous one was rolled) and 
 appends an edit - which is lost
 The following seems like a possible solution:
 1)Master detects RS#1 is dead
 2)The master renames the /hbase/.logs/regionserver name  directory to 
 something else (say /hbase/.logs/regionserver name-dead)
 3)Add mkdir support (as opposed to mkdirs) to HDFS - so that a file 
 create fails if the directory doesn't exist. Dhruba tells me this is very 
 doable.
 4)RS#1 comes back up and is not able create the new hlog. It restarts 
 itself.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138491#comment-13138491
 ] 

Ted Yu commented on HBASE-4528:
---

Using TRUNK, I wasn't able to reproduce test failure for 
TestDistributedLogSplitting:
{code}
  622  ~/runtest.sh 3 TestDistributedLogSplitting#testWorkerAbort
  623  ~/runtest.sh 3 TestDistributedLogSplitting#testWorkerAbort
  624  ~/runtest.sh 3 TestDistributedLogSplitting
{code}
The changes to HLogSplitter.java can be omitted at time of integration.

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk.txt, HBASE-4528-Trunk-FINAL.patch, 
 appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
 appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, 
 appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138497#comment-13138497
 ] 

stack commented on HBASE-4528:
--

@Ted So you are +1?

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk.txt, HBASE-4528-Trunk-FINAL.patch, 
 appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
 appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, 
 appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4680) FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have permissions

2011-10-28 Thread Gary Helmling (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling updated HBASE-4680:
-

Affects Version/s: 0.94.0

 FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have 
 permissions
 -

 Key: HBASE-4680
 URL: https://issues.apache.org/jira/browse/HBASE-4680
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.92.0, 0.94.0
Reporter: Gary Helmling
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-4680.patch


 The HDFS safe mode check workaround introduced by HBASE-4510 performs a 
 {{FileSystem.setPermission()}} operation on the root directory (/) when 
 attempting to trigger a {{SafeModeException}}.  As a result, it requires 
 superuser privileges when running with DFS permission checking enabled.  
 Changing the operations to act on the HBase root directory should be safe, 
 since the master process must have write access to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4680) FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have permissions

2011-10-28 Thread Gary Helmling (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling updated HBASE-4680:
-

  Resolution: Fixed
Assignee: Gary Helmling
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Patch committed to 0.92 branch and trunk.

 FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have 
 permissions
 -

 Key: HBASE-4680
 URL: https://issues.apache.org/jira/browse/HBASE-4680
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.92.0, 0.94.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-4680.patch


 The HDFS safe mode check workaround introduced by HBASE-4510 performs a 
 {{FileSystem.setPermission()}} operation on the root directory (/) when 
 attempting to trigger a {{SafeModeException}}.  As a result, it requires 
 superuser privileges when running with DFS permission checking enabled.  
 Changing the operations to act on the HBase root directory should be safe, 
 since the master process must have write access to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138504#comment-13138504
 ] 

Ted Yu commented on HBASE-4528:
---

All the hadoop slots on Apache Jenkins are up.
I think we should attach the final patch for TRUNK and resubmit for PreCommit 
build.

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk.txt, HBASE-4528-Trunk-FINAL.patch, 
 appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
 appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, 
 appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


 [ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4528:
--

Status: Open  (was: Patch Available)

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk.txt, HBASE-4528-Trunk-FINAL.patch, 
 appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, 
 appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, 
 appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


 [ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4528:
--

Status: Patch Available  (was: Open)

Run test suite without changes to HLogSplitter.java

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


 [ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4528:
--

Attachment: 4528-trunk-v9.txt

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut failure

2011-10-28 Thread Eugene Koontz (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koontz updated HBASE-4690:
-


Doing git bisect, it seems that both 
TestRegionServerCoprocessorExceptionWithRemove and 
TestRegionServerCoprocessorExceptionWithAbort stop working at :

306ab94... HBASE-4300 Start of new-version master fails if old master's znode 
is hanging around

 Intermittent 
 TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut
  failure
 -

 Key: HBASE-4690
 URL: https://issues.apache.org/jira/browse/HBASE-4690
 Project: HBase
  Issue Type: Test
Affects Versions: 0.92.0
Reporter: Ted Yu
Assignee: Eugene Koontz
 Fix For: 0.92.0


 See 
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
 Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
 server.
 One fix for this issue is to spin up MiniCluster with 1 region server so that 
 we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2312) Possible data loss when RS goes into GC pause while rolling HLog

2011-10-28 Thread Phabricator (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138542#comment-13138542
 ] 

Phabricator commented on HBASE-2312:


nspiegelberg has commented on the revision HBASE-2312 [jira] Possible data 
loss when RS goes into GC pause while rolling HLog.

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java:192 will 
fix.  fyi : this got auto-formatted by our apache template
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java:831 
@tedyu this patch is for 0.92.  please read the JIRA thread before this patch

REVISION DETAIL
  https://reviews.facebook.net/D99


 Possible data loss when RS goes into GC pause while rolling HLog
 

 Key: HBASE-2312
 URL: https://issues.apache.org/jira/browse/HBASE-2312
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.90.0
Reporter: Karthik Ranganathan
Assignee: Nicolas Spiegelberg
Priority: Critical
 Fix For: 0.92.0

 Attachments: D99.1.patch


 There is a very corner case when bad things could happen(ie data loss):
 1)RS #1 is going to roll its HLog - not yet created the new one, old one 
 will get no more writes
 2)RS #1 enters GC Pause of Death
 3)Master lists HLog files of RS#1 that is has to split as RS#1 is dead, 
 starts splitting
 4)RS #1 wakes up, created the new HLog (previous one was rolled) and 
 appends an edit - which is lost
 The following seems like a possible solution:
 1)Master detects RS#1 is dead
 2)The master renames the /hbase/.logs/regionserver name  directory to 
 something else (say /hbase/.logs/regionserver name-dead)
 3)Add mkdir support (as opposed to mkdirs) to HDFS - so that a file 
 create fails if the directory doesn't exist. Dhruba tells me this is very 
 doable.
 4)RS#1 comes back up and is not able create the new hlog. It restarts 
 itself.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2312) Possible data loss when RS goes into GC pause while rolling HLog


[ 
https://issues.apache.org/jira/browse/HBASE-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138554#comment-13138554
 ] 

Ted Yu commented on HBASE-2312:
---

0.92 is using hadoop 0.20.205 to build.
{code}
zhihyu$ svn log pom.xml | head

r1188404 | stack | 2011-10-24 14:57:04 -0700 (Mon, 24 Oct 2011) | 1 line

HBASE-4437 Update hadoop in 0.92 (0.20.205?)
{code}

 Possible data loss when RS goes into GC pause while rolling HLog
 

 Key: HBASE-2312
 URL: https://issues.apache.org/jira/browse/HBASE-2312
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.90.0
Reporter: Karthik Ranganathan
Assignee: Nicolas Spiegelberg
Priority: Critical
 Fix For: 0.92.0

 Attachments: D99.1.patch


 There is a very corner case when bad things could happen(ie data loss):
 1)RS #1 is going to roll its HLog - not yet created the new one, old one 
 will get no more writes
 2)RS #1 enters GC Pause of Death
 3)Master lists HLog files of RS#1 that is has to split as RS#1 is dead, 
 starts splitting
 4)RS #1 wakes up, created the new HLog (previous one was rolled) and 
 appends an edit - which is lost
 The following seems like a possible solution:
 1)Master detects RS#1 is dead
 2)The master renames the /hbase/.logs/regionserver name  directory to 
 something else (say /hbase/.logs/regionserver name-dead)
 3)Add mkdir support (as opposed to mkdirs) to HDFS - so that a file 
 create fails if the directory doesn't exist. Dhruba tells me this is very 
 doable.
 4)RS#1 comes back up and is not able create the new hlog. It restarts 
 itself.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut failure


[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138560#comment-13138560
 ] 

stack commented on HBASE-4690:
--

@Eugene Are you up to date?  The HBase-4300 signal might be false (I broke 
everything yesterday w/ that commit).  Do they fail for you?  I can't make them 
fail for me on my HW.

 Intermittent 
 TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut
  failure
 -

 Key: HBASE-4690
 URL: https://issues.apache.org/jira/browse/HBASE-4690
 Project: HBase
  Issue Type: Test
Affects Versions: 0.92.0
Reporter: Ted Yu
Assignee: Eugene Koontz
 Fix For: 0.92.0


 See 
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
 Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
 server.
 One fix for this issue is to spin up MiniCluster with 1 region server so that 
 we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut failure

2011-10-28 Thread Eugene Koontz (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138592#comment-13138592
 ] 

Eugene Koontz commented on HBASE-4690:
--

Hi Stack, I am up to date now (b04234f152a) and no failures with either 
TestRegionServerCoprocessorExceptionWithAbort or 
TestRegionServerCoprocessorExceptionWithRemove. I'm looking for any 
intermittent failures though by running a script to repeating these tests over 
and over today. 

I think you are right about consolidating these two tests - no reason to spin 
up a cluster twice. I'll open a new JIRA for that.



 Intermittent 
 TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut
  failure
 -

 Key: HBASE-4690
 URL: https://issues.apache.org/jira/browse/HBASE-4690
 Project: HBase
  Issue Type: Test
Affects Versions: 0.92.0
Reporter: Ted Yu
Assignee: Eugene Koontz
 Fix For: 0.92.0


 See 
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
 Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
 server.
 One fix for this issue is to spin up MiniCluster with 1 region server so that 
 we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-4641) Block cache can be mistakenly instantiated on Master

2011-10-28 Thread Ted Yu (Issue Comment Edited) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138126#comment-13138126
 ] 

Ted Yu edited comment on HBASE-4641 at 10/28/11 6:19 PM:
-

Looking at MiniHBaseCluster.init(), we already clone conf and give to each 
region server:
{code}
  for (int i=0; inRegionNodes; i++) {
Configuration rsConf = HBaseConfiguration.create(conf);
User user = HBaseTestingUtility.getDifferentUser(rsConf,
.hfs.+index++);
hbaseCluster.addRegionServer(rsConf, i, user);
  }
{code}
Since Configuration object isn't shared by master and region server(s) in 
minicluster, there is no need to replicate Configuration as patch v1 did.

  was (Author: yuzhih...@gmail.com):
I think the fact that Configuration object may be shared by both master and 
region server(s) in minicluster poses limitation on this and future tasks.
We should think of changing this sharing which would never happen in production 
cluster.

Once that is done, there would be no need to replicate Configuration as patch 
v1 did.
  
 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, HBASE-4641-v1.patch, 
 HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4693) Consolidate TestRegionServerCoprocessorExceptionWithRemove and TestRegionServerCoprocessorExceptionWithRemove into a single cluster spin-up

2011-10-28 Thread Eugene Koontz (Created) (JIRA)

Consolidate TestRegionServerCoprocessorExceptionWithRemove and 
TestRegionServerCoprocessorExceptionWithRemove into a single cluster spin-up
---

 Key: HBASE-4693
 URL: https://issues.apache.org/jira/browse/HBASE-4693
 Project: HBase
  Issue Type: Improvement
Reporter: Eugene Koontz
Assignee: Eugene Koontz




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4693) Consolidate TestRegionServerCoprocessorExceptionWithRemove and TestRegionServerCoprocessorExceptionWithAbort into a single cluster spin-up

2011-10-28 Thread Eugene Koontz (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koontz updated HBASE-4693:
-

Summary: Consolidate TestRegionServerCoprocessorExceptionWithRemove and 
TestRegionServerCoprocessorExceptionWithAbort into a single cluster spin-up  
(was: Consolidate TestRegionServerCoprocessorExceptionWithRemove and 
TestRegionServerCoprocessorExceptionWithRemove into a single cluster spin-up)

 Consolidate TestRegionServerCoprocessorExceptionWithRemove and 
 TestRegionServerCoprocessorExceptionWithAbort into a single cluster spin-up
 --

 Key: HBASE-4693
 URL: https://issues.apache.org/jira/browse/HBASE-4693
 Project: HBase
  Issue Type: Improvement
Reporter: Eugene Koontz
Assignee: Eugene Koontz



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4693) Consolidate TestRegionServerCoprocessorExceptionWithRemove and TestRegionServerCoprocessorExceptionWithAbort into a single cluster spin-up

2011-10-28 Thread Eugene Koontz (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eugene Koontz updated HBASE-4693:
-

Release Note:
Rather than spinning up a cluster per test, instead create one test with:

1. start cluster
2. load buggy coprocessor on one regionserserver with
hbase.coprocessor.abortonerror=false (this is the default configuration).
3. trigger coprocessor's NPE and assert that the RS doesn't abort and that the
RS removes the coprocessor.
4. restart the RS with a modified configuration with
hbase.coprocessor.abortonerror=true.
5. trigger coprocessor's NPE and assert that the RS aborts as expected.

was:
Rather than spinning up a cluster per test, instead create one test that could
be test could be:

1. start cluster
2. load buggy coprocessor on one regionserserver with
hbase.coprocessor.abortonerror=false (this is the default configuration).
3. trigger coprocessor's NPE and test that the RS doesn't abort and that the RS
removes the coprocessor.
4. restart the RS with a modified configuration with
hbase.coprocessor.abortonerror=true.
5. trigger coprocessor's NPE and test that the RS aborts as expected.

Consolidate TestRegionServerCoprocessorExceptionWithRemove and
TestRegionServerCoprocessorExceptionWithAbort into a single cluster spin-up
--

Key: HBASE-4693
URL: https://issues.apache.org/jira/browse/HBASE-4693
Project: HBase
Issue Type: Improvement
Reporter: Eugene Koontz
Assignee: Eugene Koontz

[jira] [Updated] (HBASE-4641) Block cache can be mistakenly instantiated on Master


 [ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4641:
--

Attachment: 4641-v4.txt

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, 4641-v4.txt, 
 HBASE-4641-v1.patch, HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2611) Handle RS that fails while processing the failure of another one

2011-10-28 Thread Chris Trezzo (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138637#comment-13138637
 ] 

Chris Trezzo commented on HBASE-2611:
-

I think adding the ability to atomically move a znode and all its child znodes 
might be a pretty invasive change. I couldn't seem to find any utility package 
for this on the net, but there is a patch in Zookeeper 
([ZOOKEEPER-965|https://issues.apache.org/jira/browse/ZOOKEEPER-965]) 
implementing atomic batch operations that is scheduled for 3.4.

I thought about the problem a little bit, and after conferring with Lars, I 
think we might not need the atomic move (although it would definitely make it 
simpler).

Below is some pseudo code for the algorithm I came up with. It is very similar 
to what you suggested above. Both intentions and locks are tagged with the 
region server they point to (i.e. locks are tagged with the rs that holds them, 
and intentions are tagged with the rs they intend to lock). Intentions are at 
the same level in the znode structure as locks. It is a recursive, depth first 
algorithm.

Questions/comments/suggestions always appreciated.

Chris

{code}

//this method is the top-level failover method (i.e. NodeFailoverWorker.run())
failOverRun(FailedNode a) {
  recordIntention(a, this);
  if(getLock(a, this)) {
//transfer all queues to local node
moveState(a, this, this);
  }
  else {
deleteIntention(a, this);
return;
  }
  replicateQueues();
}

moveState(NodeToMove a, CurrentNode c, TargetNode t) {
  if(lock exists on a) {
if(lock on a is owned by c) {
  moveStateHelper(a, c, t);
}
else {
  //someone else has the lock and is handling
  //the failover
  deleteIntention(a, c);
}
  }
  else {
if(queue znodes exist) {
  //we know that this node has queues to transfer
  if(getLock(a, c)) {
moveStateHelper(a, c, t);
  }
  else {
deleteIntention(a, c);
  }
}
else {
  //we know that this node is being deleted
  deleteState(a);
  deleteIntention(a, c);
}
  }
}

moveStateHelper(NodeToMove a, CurrentNode c, TargetNode t) {
  for(every intention b of a) {
moveState(b, a, t);
  }
  //we need to safely handle the case where we try to copy
  //queues that have already been copied
  copy all queues in a to t;
  deleteState(a);
  deleteIntention(a, c);
}

deleteState(NodeToDelete d) {
  //there is no need to traverse down the tree at all
  //because at this point everything below us should have
  //been deleted
  //
  //we need to safely handle the case where we attempt to delete
  //nodes that have already been deleted

  delete entire node;
}

{code}

 Handle RS that fails while processing the failure of another one
 

 Key: HBASE-2611
 URL: https://issues.apache.org/jira/browse/HBASE-2611
 Project: HBase
  Issue Type: Sub-task
  Components: replication
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans

 HBASE-2223 doesn't manage region servers that fail while doing the transfer 
 of HLogs queues from other region servers that failed. Devise a reliable way 
 to do it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4641) Block cache can be mistakenly instantiated on Master


[ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138634#comment-13138634
 ] 

Ted Yu commented on HBASE-4641:
---

Patch v4 uses the approach from comment @ 28/Oct/11 06:38

I verified through breakpoint that both CreateTableHandler and meta region 
creation are covered.
For region server, the following condition evaluates to false:
{code}
if (conf.getBoolean(HConstants.IS_MASTER, false)) {
{code}

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, 4641-v4.txt, 
 HBASE-4641-v1.patch, HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4641) Block cache can be mistakenly instantiated on Master


[ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138670#comment-13138670
 ] 

stack commented on HBASE-4641:
--

+1 on commit

Its hacky but until we break apart cache config from cache instance, this will 
do.

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, 4641-v4.txt, 
 HBASE-4641-v1.patch, HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4694) Some cleanup of log messages in RS and M

2011-10-28 Thread stack (Created) (JIRA)

Some cleanup of log messages in RS and M


 Key: HBASE-4694
 URL: https://issues.apache.org/jira/browse/HBASE-4694
 Project: HBase
  Issue Type: Improvement
Reporter: stack
 Fix For: 0.92.0


I did a little edit of logging.  We do way too much but am not going to do a 
big overhaul just yet.  Here's a few small changes saving a few lines, some 
redundancy, and making others look like surrounding log lines.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4694) Some cleanup of log messages in RS and M


 [ 
https://issues.apache.org/jira/browse/HBASE-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4694:
-

Status: Patch Available  (was: Open)

 Some cleanup of log messages in RS and M
 

 Key: HBASE-4694
 URL: https://issues.apache.org/jira/browse/HBASE-4694
 Project: HBase
  Issue Type: Improvement
Reporter: stack
 Fix For: 0.92.0

 Attachments: logging.txt


 I did a little edit of logging.  We do way too much but am not going to do a 
 big overhaul just yet.  Here's a few small changes saving a few lines, some 
 redundancy, and making others look like surrounding log lines.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4692) HBASE-4300 broke the build


 [ 
https://issues.apache.org/jira/browse/HBASE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4692:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Marking resolved since applied to branch and trunk

 HBASE-4300 broke the build
 --

 Key: HBASE-4692
 URL: https://issues.apache.org/jira/browse/HBASE-4692
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4692-v2.txt, 4692.txt


 Rebasing my patch I missed encoding of master name up in zk.  Creating a 
 separate issue so can try patch build with my fix rather than run all tests 
 locally.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4480) Testing script to simplfy local testing


 [ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4480:
--

Attachment: (was: runtest.sh)

 Testing script to simplfy local testing
 ---

 Key: HBASE-4480
 URL: https://issues.apache.org/jira/browse/HBASE-4480
 Project: HBase
  Issue Type: Improvement
Reporter: Jesse Yates
Priority: Minor
  Labels: test
 Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, 
 HBASE-4480_v3.patch, runtest2.sh


 As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
 http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
 script that would handle more of the finer points of running/checking our 
 test suite.
 This script should:
 (1) Allow people to determine which tests are hanging/taking a long time to 
 run
 (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
 running the whole suite that caused the failure
 (3) Allow people to specify to run just unit tests or also integration tests 
 (essentially wrapping calls to 'maven test' and 'maven verify').
 This script should just be a convenience script - running tests directly from 
 maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4480) Testing script to simplfy local testing


 [ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4480:
--

Attachment: runtest.sh

Modified script searches for NPE in test output and bails if NPE is found

 Testing script to simplfy local testing
 ---

 Key: HBASE-4480
 URL: https://issues.apache.org/jira/browse/HBASE-4480
 Project: HBase
  Issue Type: Improvement
Reporter: Jesse Yates
Priority: Minor
  Labels: test
 Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, 
 HBASE-4480_v3.patch, runtest.sh, runtest2.sh


 As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
 http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
 script that would handle more of the finer points of running/checking our 
 test suite.
 This script should:
 (1) Allow people to determine which tests are hanging/taking a long time to 
 run
 (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
 running the whole suite that caused the failure
 (3) Allow people to specify to run just unit tests or also integration tests 
 (essentially wrapping calls to 'maven test' and 'maven verify').
 This script should just be a convenience script - running tests directly from 
 maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut failure

2011-10-28 Thread Eugene Koontz (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138702#comment-13138702
 ] 

Eugene Koontz commented on HBASE-4690:
--

@Ted, thanks for runtest.sh, but the output is a bit misleading. 
TestRegionServerCoprocessorExceptionWithRemove and 
TestRegionServerCoprocessorExceptionWithAbort intentionally cause NPEs to test 
how the NPEs are handled. runtest.sh greps and finds the NPE and echos it, even 
if the tests pass.

 Intermittent 
 TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut
  failure
 -

 Key: HBASE-4690
 URL: https://issues.apache.org/jira/browse/HBASE-4690
 Project: HBase
  Issue Type: Test
Affects Versions: 0.92.0
Reporter: Ted Yu
Assignee: Eugene Koontz
 Fix For: 0.92.0


 See 
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
 Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
 server.
 One fix for this issue is to spin up MiniCluster with 1 region server so that 
 we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4480) Testing script to simplfy local testing


 [ 
https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4480:
--

Attachment: runtest-no-npe-check.sh

 Testing script to simplfy local testing
 ---

 Key: HBASE-4480
 URL: https://issues.apache.org/jira/browse/HBASE-4480
 Project: HBase
  Issue Type: Improvement
Reporter: Jesse Yates
Priority: Minor
  Labels: test
 Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, 
 HBASE-4480_v3.patch, runtest-no-npe-check.sh, runtest.sh, runtest2.sh


 As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and 
 http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a 
 script that would handle more of the finer points of running/checking our 
 test suite.
 This script should:
 (1) Allow people to determine which tests are hanging/taking a long time to 
 run
 (2) Allow rerunning of particular tests to make sure it wasn't an artifact of 
 running the whole suite that caused the failure
 (3) Allow people to specify to run just unit tests or also integration tests 
 (essentially wrapping calls to 'maven test' and 'maven verify').
 This script should just be a convenience script - running tests directly from 
 maven should not be impacted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut failure


[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138706#comment-13138706
 ] 

Ted Yu commented on HBASE-4690:
---

You can use runtest-no-npe-check.sh from HBASE-4480.

Alternatively if you run test suite, you should see some test failure soon:
{code}
Running 
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.406 sec
Running 
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 22.116 sec  
FAILURE!
{code}

 Intermittent 
 TestRegionServerCoprocessorExceptionWithRemove#testExceptionFromCoprocessorDuringPut
  failure
 -

 Key: HBASE-4690
 URL: https://issues.apache.org/jira/browse/HBASE-4690
 Project: HBase
  Issue Type: Test
Affects Versions: 0.92.0
Reporter: Ted Yu
Assignee: Eugene Koontz
 Fix For: 0.92.0


 See 
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
 Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
 server.
 One fix for this issue is to spin up MiniCluster with 1 region server so that 
 we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4641) Block cache can be mistakenly instantiated on Master

2011-10-28 Thread Ted Yu (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-4641:
-

Assignee: Ted Yu  (was: Jonathan Gray)

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, 4641-v4.txt, 
 HBASE-4641-v1.patch, HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-28 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138729#comment-13138729
 ] 

Jonathan Hsieh commented on HBASE-4532:
---

This seems to be checked into trunk now and there seems to be an extraneous 
System.out.println that is causing some of my tests to fail when run from 
maven (apparently maven buffers in memory instead of writing it out as a test 
is executing).

Here's the OOME that maven reports:

Exception in thread ThreadedStreamConsumer java.lang.OutOfMemoryError: Java 
heap spaceat java.util.Arrays.copyOf(Arrays.java:2882)at 
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)at
 java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)at 
java.lang.StringBuffer.append(StringBuffer.java:224)at 
org.apache.maven.surefire.report.ConsoleOutputFileReporter.writeMessage(ConsoleOutputFileReporter.java:115)at
 
org.apache.maven.surefire.report.MulticastingReporter.writeMessage(MulticastingReporter.java:101)at
 
org.apache.maven.surefire.report.TestSetRunListener.writeTestOutput(TestSetRunListener.java:99)at
 
org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:132)at
 
org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)at
 java.lang.Thread.run(Thread.java:662) man

I've attached a patch eliminates this issue.


 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4696) HRegionThriftServer

2011-10-28 Thread Prakash Khemani (Created) (JIRA)

HRegionThriftServer
---

 Key: HBASE-4696
 URL: https://issues.apache.org/jira/browse/HBASE-4696
 Project: HBase
  Issue Type: Bug
Reporter: Prakash Khemani




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-28 Thread Jonathan Hsieh (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4532:
--

Attachment: hbase-4532-remove-system.out.println.patch

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-28 Thread Jonathan Hsieh (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4532:
--

Attachment: (was: hbase-4532-remove-system.out.println.patch)

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4696) HRegionThriftServer' might have to indefinitely do redirtects

2011-10-28 Thread Prakash Khemani (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prakash Khemani updated HBASE-4696:
---

Environment:
HRegionThriftServer.getRowWithColumnsTs() redirects the request to the correct
region server if it has landed on the wrong region-server. With this approach
the smart-client will never get a NotServingRegionException and it will never
be able to invalidate its cache. It will indefinitely send the request to the
wrong region server and the wrong region server will always be redirecting it.

Either redirects should be turned off and the client should react to
NotServingRegionExceptions.

Or somehow a flag should be set in the response telling the client to refresh
its cache.
Summary: HRegionThriftServer' might have to indefinitely do redirtects
(was: HRegionThriftServer)

HRegionThriftServer' might have to indefinitely do redirtects
-

Key: HBASE-4696
URL: https://issues.apache.org/jira/browse/HBASE-4696
Project: HBase
Issue Type: Bug
Environment: HRegionThriftServer.getRowWithColumnsTs() redirects the
request to the correct region server if it has landed on the wrong
region-server. With this approach the smart-client will never get a
NotServingRegionException and it will never be able to invalidate its cache.
It will indefinitely send the request to the wrong region server and the
wrong region server will always be redirecting it.
Either redirects should be turned off and the client should react to
NotServingRegionExceptions.
Or somehow a flag should be set in the response telling the client to refresh
its cache.
Reporter: Prakash Khemani

[jira] [Updated] (HBASE-4696) HRegionThriftServer' might have to indefinitely do redirtects

2011-10-28 Thread Prakash Khemani (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Prakash Khemani updated HBASE-4696:
---

Description:
HRegionThriftServer.getRowWithColumnsTs() redirects the request to the correct
region server if it has landed on the wrong region-server. With this approach
the smart-client will never get a NotServingRegionException and it will never
be able to invalidate its cache. It will indefinitely send the request to the
wrong region server and the wrong region server will always be redirecting it.

Either redirects should be turned off and the client should react to
NotServingRegionExceptions.

Or somehow a flag should be set in the response telling the client to refresh
its cache.
Environment: (was: HRegionThriftServer.getRowWithColumnsTs() redirects
the request to the correct region server if it has landed on the wrong
region-server. With this approach the smart-client will never get a
NotServingRegionException and it will never be able to invalidate its cache. It
will indefinitely send the request to the wrong region server and the wrong
region server will always be redirecting it.

Either redirects should be turned off and the client should react to
NotServingRegionExceptions.

Or somehow a flag should be set in the response telling the client to refresh
its cache.)

HRegionThriftServer' might have to indefinitely do redirtects
-

Key: HBASE-4696
URL: https://issues.apache.org/jira/browse/HBASE-4696
Project: HBase
Issue Type: Bug
Reporter: Prakash Khemani

HRegionThriftServer.getRowWithColumnsTs() redirects the request to the
correct region server if it has landed on the wrong region-server. With this
approach the smart-client will never get a NotServingRegionException and it
will never be able to invalidate its cache. It will indefinitely send the
request to the wrong region server and the wrong region server will always be
redirecting it.
Either redirects should be turned off and the client should react to
NotServingRegionExceptions.
Or somehow a flag should be set in the response telling the client to refresh
its cache.

[jira] [Commented] (HBASE-2611) Handle RS that fails while processing the failure of another one


[ 
https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138739#comment-13138739
 ] 

Ted Yu commented on HBASE-2611:
---

In moveState(), if lock on a is owned by c, should lock be released after 
moveStateHelper() returns ?
I guess lock release can also be done at the end of moveStateHelper().

 Handle RS that fails while processing the failure of another one
 

 Key: HBASE-2611
 URL: https://issues.apache.org/jira/browse/HBASE-2611
 Project: HBase
  Issue Type: Sub-task
  Components: replication
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans

 HBASE-2223 doesn't manage region servers that fail while doing the transfer 
 of HLogs queues from other region servers that failed. Devise a reliable way 
 to do it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4692) HBASE-4300 broke the build


[ 
https://issues.apache.org/jira/browse/HBASE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138741#comment-13138741
 ] 

Hudson commented on HBASE-4692:
---

Integrated in HBase-TRUNK #2379 (See 
[https://builds.apache.org/job/HBase-TRUNK/2379/])
HBASE-4692 HBASE-4300 broke the build

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ServerName.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Addressing.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestServerName.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java


 HBASE-4300 broke the build
 --

 Key: HBASE-4692
 URL: https://issues.apache.org/jira/browse/HBASE-4692
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4692-v2.txt, 4692.txt


 Rebasing my patch I missed encoding of master name up in zk.  Creating a 
 separate issue so can try patch build with my fix rather than run all tests 
 locally.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1744) Thrift server to match the new java api.


[ 
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138740#comment-13138740
 ] 

Jonathan Gray commented on HBASE-1744:
--

One more requested change.  Over in HBASE-4658 the map of attributes was added 
to the available APIs in thrift.  Could we add this to the new TScan, TGet, 
etc. structs?

 Thrift server to match the new java api.
 

 Key: HBASE-1744
 URL: https://issues.apache.org/jira/browse/HBASE-1744
 Project: HBase
  Issue Type: Improvement
  Components: thrift
Reporter: Tim Sell
Assignee: Tim Sell
Priority: Critical
 Fix For: 0.94.0

 Attachments: 
 0001-thrift2-enable-usage-of-.deleteColumns-for-thrift.patch, 
 HBASE-1744.2.patch, HBASE-1744.3.patch, HBASE-1744.4.patch, 
 HBASE-1744.5.patch, HBASE-1744.6.patch, HBASE-1744.7.patch, 
 HBASE-1744.8.patch, HBASE-1744.9.patch, HBASE-1744.preview.1.patch, 
 thriftexperiment.patch


 This mutateRows, etc.. is a little confusing compared to the new cleaner java 
 client.
 Thinking of ways to make a thrift client that is just as elegant. something 
 like:
 void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
 with:
 struct TColumn {
   1:Bytes family,
   2:Bytes qualifier,
   3:i64 timestamp
 }
 struct TPut {
   1:Bytes row,
   2:mapTColumn, Bytes values
 }
 This creates more verbose rpc  than if the columns in TPut were just 
 mapBytes, mapBytes, Bytes, but that is harder to fit timestamps into and 
 still be intuitive from say python.
 Presumably the goal of a thrift gateway is to be easy first.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around


[ 
https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138742#comment-13138742
 ] 

Hudson commented on HBASE-4300:
---

Integrated in HBase-TRUNK #2379 (See 
[https://builds.apache.org/job/HBase-TRUNK/2379/])
HBASE-4692 HBASE-4300 broke the build

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ServerName.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Addressing.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestServerName.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java


 Start of new-version master fails if old master's znode is hanging around
 -

 Key: HBASE-4300
 URL: https://issues.apache.org/jira/browse/HBASE-4300
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: stack
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4300-v2.txt, 4300-v3.txt, 4300.txt


 I shut down an 0.90 cluster, and had to do so uncleanly. I then started a 
 trunk (0.92) cluster before the old master znode had expired. This cased:
 java.lang.StringIndexOutOfBoundsException: String index out of range: -1
 at java.lang.String.substring(String.java:1937)
 at 
 org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81)
 at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63)
 at 
 org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148)
 at 
 org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4680) FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have permissions


[ 
https://issues.apache.org/jira/browse/HBASE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138743#comment-13138743
 ] 

Hudson commented on HBASE-4680:
---

Integrated in HBase-TRUNK #2379 (See 
[https://builds.apache.org/job/HBase-TRUNK/2379/])
HBASE-4680  FSUtils.isInSafeMode() checks should operate on HBase root dir, 
where we have permissions

garyh : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java


 FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have 
 permissions
 -

 Key: HBASE-4680
 URL: https://issues.apache.org/jira/browse/HBASE-4680
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.92.0, 0.94.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical
 Fix For: 0.92.0

 Attachments: HBASE-4680.patch


 The HDFS safe mode check workaround introduced by HBASE-4510 performs a 
 {{FileSystem.setPermission()}} operation on the root directory (/) when 
 attempting to trigger a {{SafeModeException}}.  As a result, it requires 
 superuser privileges when running with DFS permission checking enabled.  
 Changing the operations to act on the HBase root directory should be safe, 
 since the master process must have write access to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4304) requestsPerSecond counter stuck at 0

2011-10-28 Thread jirapos...@reviews.apache.org (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138744#comment-13138744
 ] 

Hudson commented on HBASE-4304:
---

Integrated in HBase-TRUNK #2379 (See 
[https://builds.apache.org/job/HBase-TRUNK/2379/])
HBASE-4304 requestsPerSecond counter stuck at 0

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/MetricsRate.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsMBeanBase.java


 requestsPerSecond counter stuck at 0
 

 Key: HBASE-4304
 URL: https://issues.apache.org/jira/browse/HBASE-4304
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Li Pi
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4304v1.txt, 4304v2.txt, 4304v3.txt, 4304v4.txt


 Running trunk @ r1163343, all of the requestsPerSecond counters are showing 0 
 both in the master UI and in the RS UI. The writeRequestsCount metric is 
 properly updating in the RS UI.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4687) regionserver may miss zk-heartbeats to master when replaying edits at region open


[ 
https://issues.apache.org/jira/browse/HBASE-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138752#comment-13138752
 ] 

jirapos...@reviews.apache.org commented on HBASE-4687:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2585/#review2916
---

Ship it!


looks good to me

- Jonathan


On 2011-10-27 19:32:54, Prakash Khemani wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2585/
bq.  ---
bq.  
bq.  (Updated 2011-10-27 19:32:54)
bq.  
bq.  
bq.  Review request for hbase, Dhruba Borthakur, Jonathan Gray, and Nicolas 
Spiegelberg.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  must report.progress() at least once per edits file when replaying edits
bq.  
bq.  
bq.  This addresses bug HBASE-4687.
bq.  https://issues.apache.org/jira/browse/HBASE-4687
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 5243b58 
bq.  
bq.  Diff: https://reviews.apache.org/r/2585/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  small change and not tested.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Prakash
bq.  
bq.



 regionserver may miss zk-heartbeats to master when replaying edits at region 
 open
 -

 Key: HBASE-4687
 URL: https://issues.apache.org/jira/browse/HBASE-4687
 Project: HBase
  Issue Type: Bug
Reporter: Prakash Khemani
Assignee: Prakash Khemani

 replayRecoveredEdits() should do another reporter.progress() before returning.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1744) Thrift server to match the new java api.

2011-10-28 Thread Tim Sell (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138751#comment-13138751
 ] 

Tim Sell commented on HBASE-1744:
-

I think that should be a follow up issue to this rather than a block.

 Thrift server to match the new java api.
 

 Key: HBASE-1744
 URL: https://issues.apache.org/jira/browse/HBASE-1744
 Project: HBase
  Issue Type: Improvement
  Components: thrift
Reporter: Tim Sell
Assignee: Tim Sell
Priority: Critical
 Fix For: 0.94.0

 Attachments: 
 0001-thrift2-enable-usage-of-.deleteColumns-for-thrift.patch, 
 HBASE-1744.2.patch, HBASE-1744.3.patch, HBASE-1744.4.patch, 
 HBASE-1744.5.patch, HBASE-1744.6.patch, HBASE-1744.7.patch, 
 HBASE-1744.8.patch, HBASE-1744.9.patch, HBASE-1744.preview.1.patch, 
 thriftexperiment.patch


 This mutateRows, etc.. is a little confusing compared to the new cleaner java 
 client.
 Thinking of ways to make a thrift client that is just as elegant. something 
 like:
 void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
 with:
 struct TColumn {
   1:Bytes family,
   2:Bytes qualifier,
   3:i64 timestamp
 }
 struct TPut {
   1:Bytes row,
   2:mapTColumn, Bytes values
 }
 This creates more verbose rpc  than if the columns in TPut were just 
 mapBytes, mapBytes, Bytes, but that is harder to fit timestamps into and 
 still be intuitive from say python.
 Presumably the goal of a thrift gateway is to be easy first.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4696) HRegionThriftServer' might have to indefinitely do redirtects

2011-10-28 Thread Jonathan Gray (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray reassigned HBASE-4696:


Assignee: Jonathan Gray

 HRegionThriftServer' might have to indefinitely do redirtects
 -

 Key: HBASE-4696
 URL: https://issues.apache.org/jira/browse/HBASE-4696
 Project: HBase
  Issue Type: Bug
Reporter: Prakash Khemani
Assignee: Jonathan Gray

 HRegionThriftServer.getRowWithColumnsTs() redirects the request to the 
 correct region server if it has landed on the wrong region-server. With this 
 approach the smart-client will never get a NotServingRegionException and it 
 will never be able to invalidate its cache. It will indefinitely send the 
 request to the wrong region server and the wrong region server will always be 
 redirecting it.
 Either redirects should be turned off and the client should react to 
 NotServingRegionExceptions.
 Or somehow a flag should be set in the response telling the client to refresh 
 its cache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

2011-10-28 Thread Hadoop QA (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138756#comment-13138756
]

Hadoop QA commented on HBASE-4528:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501323/4528-trunk-v9.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 18 new or modified tests.

-1 javadoc. The javadoc tool appears to have generated -167 warning
messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint

org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/87//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/87//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/87//console

This message is automatically generated.

The put operation can release the rowlock before sync-ing the Hlog
--

Key: HBASE-4528
URL: https://issues.apache.org/jira/browse/HBASE-4528
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Fix For: 0.94.0

Attachments: 4528-trunk-v9.txt, 4528-trunk.txt,
HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt,
appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt,
appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt,
appendNoSyncPut8.txt

This allows for better throughput when there are hot rows. A single row
update improves from 100 puts/sec/server to 5000 puts/sec/server.

[jira] [Commented] (HBASE-2611) Handle RS that fails while processing the failure of another one

2011-10-28 Thread Chris Trezzo (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138762#comment-13138762
 ] 

Chris Trezzo commented on HBASE-2611:
-

I should have specified that in deleteState(), the line delete entire node 
deletes the entire znode replication hierarchy for that region server. This 
would include the lock znode, which is essentially releasing the lock at the 
end of moveStateHelper().

Thanks!
Chris

 Handle RS that fails while processing the failure of another one
 

 Key: HBASE-2611
 URL: https://issues.apache.org/jira/browse/HBASE-2611
 Project: HBase
  Issue Type: Sub-task
  Components: replication
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans

 HBASE-2223 doesn't manage region servers that fail while doing the transfer 
 of HLogs queues from other region servers that failed. Devise a reliable way 
 to do it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4696) HRegionThriftServer' might have to indefinitely do redirtects

[
https://issues.apache.org/jira/browse/HBASE-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Gray updated HBASE-4696:
-

Attachment: HBASE-4696-v1.patch

Adds new redirect configuration option to HRegionThriftServer:
{{hbase.regionserver.thrift.redirect}}

Default is false.

If this is false, then if the current regionserver does not have the specified
region, an exception is thrown rather than auto-re-directing.

This also has another fix where it was using the encoded region name instead of
the full binary region name.

HRegionThriftServer' might have to indefinitely do redirtects
-

Key: HBASE-4696
URL: https://issues.apache.org/jira/browse/HBASE-4696
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Prakash Khemani
Assignee: Jonathan Gray
Fix For: 0.94.0

Attachments: HBASE-4696-v1.patch

[jira] [Updated] (HBASE-4696) HRegionThriftServer' might have to indefinitely do redirtects


 [ 
https://issues.apache.org/jira/browse/HBASE-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray updated HBASE-4696:
-

Fix Version/s: 0.94.0
Affects Version/s: 0.94.0
 Release Note: RS embedded thrift server can have redirecting turned 
on/off with hbase.regionserver.thrift.redirect (default: false)
   Status: Patch Available  (was: Open)

 HRegionThriftServer' might have to indefinitely do redirtects
 -

 Key: HBASE-4696
 URL: https://issues.apache.org/jira/browse/HBASE-4696
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Prakash Khemani
Assignee: Jonathan Gray
 Fix For: 0.94.0

 Attachments: HBASE-4696-v1.patch


 HRegionThriftServer.getRowWithColumnsTs() redirects the request to the 
 correct region server if it has landed on the wrong region-server. With this 
 approach the smart-client will never get a NotServingRegionException and it 
 will never be able to invalidate its cache. It will indefinitely send the 
 request to the wrong region server and the wrong region server will always be 
 redirecting it.
 Either redirects should be turned off and the client should react to 
 NotServingRegionExceptions.
 Or somehow a flag should be set in the response telling the client to refresh 
 its cache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4691) Remove more unnecessary byte[] copies from KeyValues

2011-10-28 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138785#comment-13138785
 ] 

Lars Hofhansl commented on HBASE-4691:
--

All tests pass... Will commit to trunk later today.

 Remove more unnecessary byte[] copies from KeyValues
 

 Key: HBASE-4691
 URL: https://issues.apache.org/jira/browse/HBASE-4691
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
 Fix For: 0.94.0

 Attachments: 4691.txt


 Just looking through the code I found some more spots where we unnecessarily 
 copy byte[] rather than just passing offset and length around.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


 [ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4528:
--

Attachment: (was: 4528-trunk-v9.txt)

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4683) Create config option to only cache index blocks

2011-10-28 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138787#comment-13138787
 ] 

Lars Hofhansl commented on HBASE-4683:
--

passes all test

 Create config option to only cache index blocks
 ---

 Key: HBASE-4683
 URL: https://issues.apache.org/jira/browse/HBASE-4683
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Priority: Minor
 Fix For: 0.94.0

 Attachments: 4683.txt


 This would add a new boolean config option: hfile.block.cache.datablocks
 Default would be true.
 Setting this to false allows HBase in a mode where only index blocks are 
 cached, which is useful for analytical scenarios where a useful working set 
 of the data cannot be expected to fit into the (aggregate) cache.
 This is the equivalent of setting cacheBlocks to false on all scans 
 (including scans on behalf of gets).
 I would like to get a general feeling about what folks think about this.
 The change itself would be simple.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


 [ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4528:
--

Attachment: 4528-trunk-v9.txt

Remove extraneous changes for pom.xml

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4696) HRegionThriftServer' might have to indefinitely do redirtects


 [ 
https://issues.apache.org/jira/browse/HBASE-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray updated HBASE-4696:
-

Issue Type: Improvement  (was: Bug)

Marking as improvement since this is really further improvement to a new 
feature rather than a real bug fix.

 HRegionThriftServer' might have to indefinitely do redirtects
 -

 Key: HBASE-4696
 URL: https://issues.apache.org/jira/browse/HBASE-4696
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Prakash Khemani
Assignee: Jonathan Gray
 Fix For: 0.94.0

 Attachments: HBASE-4696-v1.patch


 HRegionThriftServer.getRowWithColumnsTs() redirects the request to the 
 correct region server if it has landed on the wrong region-server. With this 
 approach the smart-client will never get a NotServingRegionException and it 
 will never be able to invalidate its cache. It will indefinitely send the 
 request to the wrong region server and the wrong region server will always be 
 redirecting it.
 Either redirects should be turned off and the client should react to 
 NotServingRegionExceptions.
 Or somehow a flag should be set in the response telling the client to refresh 
 its cache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4641) Block cache can be mistakenly instantiated on Master

[
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138792#comment-13138792
]

Jonathan Gray commented on HBASE-4641:
--

I like v4 much less than the other changes. My v1 patch makes it so we could
potentially break something because it's expecting to be able to manipulate the
conf after construction (an easy assumption to document / test for). The v4
patch now takes the conf passed in by reference and modifies it. It then
modifies the same conf reference later in Store. Seems like this could have
some bad side-effects in the opposite direction.

At this point, I vote for the v1 hack until we make the cache non-static. As
long as unit tests still pass.

Block cache can be mistakenly instantiated on Master

Key: HBASE-4641
URL: https://issues.apache.org/jira/browse/HBASE-4641
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Ted Yu
Priority: Critical
Fix For: 0.92.0

Attachments: 4641-suggestion-v3.txt, 4641-v4.txt,
HBASE-4641-v1.patch, HBASE-4641-v2.patch

After changes in the block cache instantiation over in HBASE-4422, it looks
like the HMaster can now end up with a block cache instantiated. Not a huge
deal but prevents the process from shutting down properly.

[jira] [Updated] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-28 Thread Jonathan Hsieh (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4532:
--

Attachment: hbase-4532-remove-system.out.println.patch

Attached file has the system out line removed.

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4641) Block cache can be mistakenly instantiated on Master


 [ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4641:
--

Assignee: Jonathan Gray  (was: Ted Yu)

Patch v1 is fine with me.

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, 4641-v4.txt, 
 HBASE-4641-v1.patch, HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138797#comment-13138797
 ] 

Ted Yu commented on HBASE-4528:
---

I ran the 4 failed tests reported by HadoopQA on MacBook and they all passed.
More importantly 
https://builds.apache.org/job/PreCommit-HBASE-Build/87//testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testOrphanLogCreation/
 doesn't show NPE.

I think 4528-trunk-v9.txt should be good to go.

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-28 Thread Liyin Tang (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138799#comment-13138799
 ] 

Liyin Tang commented on HBASE-4532:
---

Thanks Jonathan for the patch. I should remove this line out.

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4641) Block cache can be mistakenly instantiated on Master


[ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138800#comment-13138800
 ] 

Ted Yu commented on HBASE-4641:
---

One benefit for patch v4 is that it allows different components to make 
decisions based on whether JVM is for master or region server.
We can revisit this topic when more config parameters are set in this regard.

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, 4641-v4.txt, 
 HBASE-4641-v1.patch, HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4641) Block cache can be mistakenly instantiated on Master

2011-10-28 Thread Jonathan Gray (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray resolved HBASE-4641.
--

  Resolution: Fixed
Hadoop Flags: Reviewed

Committed v1 to branch and trunk.  Thanks guys.

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, 4641-v4.txt, 
 HBASE-4641-v1.patch, HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4697) Block cache reference should not be static

2011-10-28 Thread Jonathan Gray (Created) (JIRA)

Block cache reference should not be static
--

 Key: HBASE-4697
 URL: https://issues.apache.org/jira/browse/HBASE-4697
 Project: HBase
  Issue Type: Improvement
  Components: io
Reporter: Jonathan Gray
 Fix For: 0.94.0


After the introduction of CacheConfig, over in HBASE-4641 we dealt with a bug 
of the block cache being instantiated in the master.  A short-term fix was done 
there.

This is the real fix.  The block cache should not be a static reference, it 
should be instantiated within HRS construction/initialization.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4641) Block cache can be mistakenly instantiated on Master


[ 
https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138809#comment-13138809
 ] 

Jonathan Gray commented on HBASE-4641:
--

Opened HBASE-4697 to deal with real solution.

 Block cache can be mistakenly instantiated on Master
 

 Key: HBASE-4641
 URL: https://issues.apache.org/jira/browse/HBASE-4641
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Priority: Critical
 Fix For: 0.92.0

 Attachments: 4641-suggestion-v3.txt, 4641-v4.txt, 
 HBASE-4641-v1.patch, HBASE-4641-v2.patch


 After changes in the block cache instantiation over in HBASE-4422, it looks 
 like the HMaster can now end up with a block cache instantiated.  Not a huge 
 deal but prevents the process from shutting down properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4649) Add atomic bulk load function to region server

2011-10-28 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138813#comment-13138813
 ] 

Jonathan Hsieh commented on HBASE-4649:
---

There was a extra System.out.println introduced by HBASE-4532 that was causing 
the new unit tests to fail when run from mvn (worked fine from eclipse or via 
junit's test runner ' bin/hbase org.junit.runner.JUnitCore 
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad')

I've attached patch there.


 Add atomic bulk load function to region server
 --

 Key: HBASE-4649
 URL: https://issues.apache.org/jira/browse/HBASE-4649
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Affects Versions: 0.90.4, 0.92.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 
 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch, 
 hbase-4649.v2.patch


 Add a method that atomically bulk load multiple hfiles.  Row atomicity 
 guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

2011-10-28 Thread Ted Yu (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-4532.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

Integrated addendum to TRUNK.

Thanks Jonathan.

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


 [ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4532:
--

Fix Version/s: 0.94.0

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4687) regionserver may miss zk-heartbeats to master when replaying edits at region open

2011-10-28 Thread Prakash Khemani (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash Khemani updated HBASE-4687:
---

Attachment: 0001-HBASE-4687-regionserver-may-miss-zk-heartbeats-to-ma.patch

path attached

 regionserver may miss zk-heartbeats to master when replaying edits at region 
 open
 -

 Key: HBASE-4687
 URL: https://issues.apache.org/jira/browse/HBASE-4687
 Project: HBase
  Issue Type: Bug
Reporter: Prakash Khemani
Assignee: Prakash Khemani
 Attachments: 
 0001-HBASE-4687-regionserver-may-miss-zk-heartbeats-to-ma.patch


 replayRecoveredEdits() should do another reporter.progress() before returning.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog


 [ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray updated HBASE-4528:
-

  Resolution: Fixed
Release Note: Adds early-lock-release ability so we can do the WAL sync 
outside of the row lock.  Extends MemStore/RWCC to support rollback.
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to trunk.  Thanks Dhruba for the awesome work, Thanks Ted and others 
for all the good reviews.

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138822#comment-13138822
 ] 

Jonathan Gray commented on HBASE-4532:
--

Please stop doing multiple commits on the same JIRA! :)  I thought we agreed on 
this, or no?

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4687) regionserver may miss zk-heartbeats to master when replaying edits at region open

2011-10-28 Thread Jonathan Gray (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Gray resolved HBASE-4687.
--

   Resolution: Fixed
Fix Version/s: 0.92.0
 Hadoop Flags: Reviewed

Committed to 92 branch and trunk.

 regionserver may miss zk-heartbeats to master when replaying edits at region 
 open
 -

 Key: HBASE-4687
 URL: https://issues.apache.org/jira/browse/HBASE-4687
 Project: HBase
  Issue Type: Bug
Reporter: Prakash Khemani
Assignee: Prakash Khemani
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4687-regionserver-may-miss-zk-heartbeats-to-ma.patch


 replayRecoveredEdits() should do another reporter.progress() before returning.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4687) regionserver may miss zk-heartbeats to master when replaying edits at region open


[ 
https://issues.apache.org/jira/browse/HBASE-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138828#comment-13138828
 ] 

Jonathan Gray commented on HBASE-4687:
--

Thanks Prakash!

 regionserver may miss zk-heartbeats to master when replaying edits at region 
 open
 -

 Key: HBASE-4687
 URL: https://issues.apache.org/jira/browse/HBASE-4687
 Project: HBase
  Issue Type: Bug
Reporter: Prakash Khemani
Assignee: Prakash Khemani
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4687-regionserver-may-miss-zk-heartbeats-to-ma.patch


 replayRecoveredEdits() should do another reporter.progress() before returning.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138826#comment-13138826
 ] 

Ted Yu commented on HBASE-4532:
---

This JIRA wasn't closed before applying the addendum.
May I ask why this JIRA was integrated on the 24th without announcement ?

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138831#comment-13138831
 ] 

Jonathan Gray commented on HBASE-4532:
--

I don't think JIRA being open/closed is the issue, it's more multiple commits.

But yeah, as a separate note, looks like there was no final comment and 
resolution after the commit.

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138834#comment-13138834
 ] 

Ted Yu commented on HBASE-4532:
---

I would interpret the JIRA not being resolved as anticipation for an addendum 
:-)

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.94.0

 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog

2011-10-28 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13138846#comment-13138846
 ] 

Lars Hofhansl commented on HBASE-4528:
--

yeah! This is awesome.

 The put operation can release the rowlock before sync-ing the Hlog
 --

 Key: HBASE-4528
 URL: https://issues.apache.org/jira/browse/HBASE-4528
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Fix For: 0.94.0

 Attachments: 4528-trunk-v9.txt, 4528-trunk.txt, 
 HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, 
 appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, 
 appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, 
 appendNoSyncPut8.txt


 This allows for better throughput when there are hot rows. A single row 
 update improves from 100 puts/sec/server to 5000 puts/sec/server.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4063) Improve TableInputFormat to allow application to configure the number of mappers