date:20131017


 [ 
https://issues.apache.org/jira/browse/HBASE-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl resolved HBASE-9753.
--

  Resolution: Fixed
Hadoop Flags: Reviewed

 Excessive readpoint checks in MemstoreScanner
 -

 Key: HBASE-9753
 URL: https://issues.apache.org/jira/browse/HBASE-9753
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0, 0.94.13, 0.96.1

 Attachments: 9753-0.94.txt, 9753-v1.txt


 Brought up by [~vrodionov] on the mailing list. See also HBASE-9751.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size


[ 
https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797680#comment-13797680
 ] 

Lars Hofhansl commented on HBASE-9769:
--

Did some profiling on why reseek() is so much slower than next() even when 
reseek just has to seek to the next key. The reason is all the compares we're 
doing... For each reseek:
* 2 KV compares in KeyValueHeap.generalizedSeek to find the top scanner
* 2 key compares in HFileReaderV2.ScannerV2.reseekTo (one to check for reseek, 
one to check against the index key)
* 2 key compares in HFileReaderV2.ScannerV2.blockSeek to find the right key

After all that we finally read the KV we found.

While next() just reads the next KV from the current HFile block.

Nothing jumps here as to how we could simplify this.

 Improve performance of a Scanner with explicit column list when rows are 
 small/medium size
 --

 Key: HBASE-9769
 URL: https://issues.apache.org/jira/browse/HBASE-9769
 Project: HBase
  Issue Type: Improvement
  Components: Scanners
Affects Versions: 0.98.0, 0.94.12, 0.96.0
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, 
 9769-0.94-sample.txt, 9769-94.txt, 9769-94-v2.txt, 9769-trunk-v1.txt






--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9747) PrefixFilter with OR condition gives wrong results


[ 
https://issues.apache.org/jira/browse/HBASE-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797682#comment-13797682
 ] 

Lars Hofhansl commented on HBASE-9747:
--

Looks good to me. Will commit tomorrow, unless hear objections.

 PrefixFilter with OR condition gives wrong results
 --

 Key: HBASE-9747
 URL: https://issues.apache.org/jira/browse/HBASE-9747
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 0.95.2, 0.94.9
Reporter: Deepa Remesh
Assignee: Aditya Kishore
 Attachments: HBASE-9747_0.94.patch, HBASE-9747.patch


 PrefixFilter when used with a SingleColumnValueFilter with an OR condition 
 gives wrong results. In below example, each filter when evaluated separately 
 gives 1 row each. The OR condition with the two filters gives 3 rows instead 
 of 2. Repro below:
 create 't1', 'f1'
 put 't1','a1','f1:q2','111'
 put 't1','b1','f1:q1','112'
 put 't1','c1','f1:q1','113'
 hbase(main):020:0 scan 't1', {FILTER = PrefixFilter ('b') OR 
 SingleColumnValueFilter('f1', 'q1', =, 'binary:113')}
 ROWCOLUMN+CELL
  a1column=f1:q2, 
 timestamp=1381468905492, value=111
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
  c1column=f1:q1, 
 timestamp=1381468905549, value=113
 3 row(s) in 0.1020 seconds
 hbase(main):021:0 scan 't1', {FILTER = PrefixFilter ('b')}
 ROWCOLUMN+CELL
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
 1 row(s) in 0.0150 seconds
 hbase(main):002:0 scan 't1', {FILTER = SingleColumnValueFilter('f1', 'q1', 
 =, 'binary:113')}
 ROWCOLUMN+CELL
  c1column=f1:q1, 
 timestamp=1381469178679, value=113
 1 row(s) in 0.0140 seconds



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9720) TestSplitTransactionOnCluster#testShutdownFixupWhenDaughterHasSplit occasionally times out


[ 
https://issues.apache.org/jira/browse/HBASE-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797687#comment-13797687
 ] 

Hudson commented on HBASE-9720:
---

SUCCESS: Integrated in hbase-0.96 #145 (See 
[https://builds.apache.org/job/hbase-0.96/145/])
HBASE-9720 TestSplitTransactionOnCluster#testShutdownFixupWhenDaughterHasSplit 
occasionally times out (tedyu: rev 1532962)
* 
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java


 TestSplitTransactionOnCluster#testShutdownFixupWhenDaughterHasSplit 
 occasionally times out
 --

 Key: HBASE-9720
 URL: https://issues.apache.org/jira/browse/HBASE-9720
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
 Fix For: 0.98.0, 0.96.1

 Attachments: 9720-v1.txt


 From 
 https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/779/testReport/junit/org.apache.hadoop.hbase.regionserver/TestSplitTransactionOnCluster/testShutdownFixupWhenDaughterHasSplit/
  :
 {code}
 java.lang.AssertionError: Waited too long for split
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at org.junit.Assert.assertFalse(Assert.java:64)
   at 
 org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.split(TestSplitTransactionOnCluster.java:1065)
   at 
 org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testShutdownFixupWhenDaughterHasSplit(TestSplitTransactionOnCluster.java:442)
 ...
 2013-10-05 13:00:18,060 DEBUG 
 [RS:2;quirinus:46584-smallCompactions-1380978003766] 
 regionserver.HRegionFileSystem(338): Committing store file 
 hdfs://localhost:45166/user/jenkins/hbase/data/default/testShutdownFixupWhenDaughterHasSplit/0d7218d1ce3bd629779009821908a3ed/.tmp/8b155b635b304a368e11dbd675d09312
  as 
 hdfs://localhost:45166/user/jenkins/hbase/data/default/testShutdownFixupWhenDaughterHasSplit/0d7218d1ce3bd629779009821908a3ed/info/8b155b635b304a368e11dbd675d09312
 2013-10-05 13:00:18,436 DEBUG [pool-1-thread-1-EventThread] 
 zookeeper.ZooKeeperWatcher(310): master:48355-0x14188b3d794 Received 
 ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/balancer
 2013-10-05 13:00:18,436 DEBUG [AM.ZK.Worker-pool2-t11] 
 master.AssignmentManager(818): Handling RS_ZK_REGION_SPLITTING, 
 server=quirinus.apache.org,46584,1380977990795, 
 region=e29b00c3bdaa3e10f6c4fe252a82399f, 
 current_state={e29b00c3bdaa3e10f6c4fe252a82399f state=SPLITTING, 
 ts=1380978012468, server=quirinus.apache.org,46584,1380977990795}
 {code}
 We can see the following stack traces at the end of test output:
 {code}
 Potentially hanging thread: RS:2;quirinus:46584-smallCompactions-1380978003766
   java.lang.Object.wait(Native Method)
   java.lang.Object.wait(Object.java:485)
   org.apache.hadoop.ipc.Client.call(Client.java:1333)
   org.apache.hadoop.ipc.Client.call(Client.java:1300)
   
 org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
   com.sun.proxy.$Proxy17.rename(Unknown Source)
   sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
   
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   java.lang.reflect.Method.invoke(Method.java:597)
   
 org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:188)
   
 org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
   com.sun.proxy.$Proxy17.rename(Unknown Source)
   
 org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.rename(ClientNamenodeProtocolTranslatorPB.java:396)
   sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   java.lang.reflect.Method.invoke(Method.java:597)
   org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:266)
   com.sun.proxy.$Proxy22.rename(Unknown Source)
   org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:1512)
   
 org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:528)
   org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:210)
   
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.rename(HRegionFileSystem.java:924)
   
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:340)
   
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:312)
   
 org.apache.hadoop.hbase.regionserver.HStore.moveFileIntoPlace(HStore.java:1032)

[jira] [Updated] (HBASE-9689) Support using OperationAttributes through shell script


 [ 
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-9689:
--

Status: Open  (was: Patch Available)

 Support using OperationAttributes through shell script
 --

 Key: HBASE-9689
 URL: https://issues.apache.org/jira/browse/HBASE-9689
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-9689_1.patch, HBASE-9689_2.patch, 
 HBASE-9689_3.patch, HBASE-9689.patch


 Currently the ruby scripts for Put does not support setting of Operation 
 Attributes through shell.  
 It may be useful in some cases and also for testing.  And that would give a 
 full  fledged support using shell.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9689) Support using OperationAttributes through shell script


 [ 
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-9689:
--

Attachment: HBASE-9689_3.patch

This patch works as expected.  No backward compatibility changes.  TestShell 
passes with this.  
If it is fine I can commit this later today.

 Support using OperationAttributes through shell script
 --

 Key: HBASE-9689
 URL: https://issues.apache.org/jira/browse/HBASE-9689
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-9689_1.patch, HBASE-9689_2.patch, 
 HBASE-9689_3.patch, HBASE-9689.patch


 Currently the ruby scripts for Put does not support setting of Operation 
 Attributes through shell.  
 It may be useful in some cases and also for testing.  And that would give a 
 full  fledged support using shell.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9689) Support using OperationAttributes through shell script


 [ 
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-9689:
--

Status: Patch Available  (was: Open)

 Support using OperationAttributes through shell script
 --

 Key: HBASE-9689
 URL: https://issues.apache.org/jira/browse/HBASE-9689
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-9689_1.patch, HBASE-9689_2.patch, 
 HBASE-9689_3.patch, HBASE-9689.patch


 Currently the ruby scripts for Put does not support setting of Operation 
 Attributes through shell.  
 It may be useful in some cases and also for testing.  And that would give a 
 full  fledged support using shell.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-7663) [Per-KV security] Visibility labels


 [ 
https://issues.apache.org/jira/browse/HBASE-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-7663:
--

Attachment: HBASE-7663.patch

 [Per-KV security] Visibility labels
 ---

 Key: HBASE-7663
 URL: https://issues.apache.org/jira/browse/HBASE-7663
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, security
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Anoop Sam John
 Attachments: HBASE-7663.patch


 Implement Accumulo-style visibility labels. Consider the following design 
 principles:
 - Coprocessor based implementation
 - Minimal to no changes to core code
 - Use KeyValue tags (HBASE-7448) to carry labels
 - Use OperationWithAttributes# {get,set}Attribute for handling visibility 
 labels in the API
 - Implement a new filter for evaluating visibility labels as KVs are streamed 
 through.
 This approach would be consistent in deployment and API details with other 
 per-KV security work, supporting environments where they might be both be 
 employed, even stacked on some tables.
 See the parent issue for more discussion.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-7663) [Per-KV security] Visibility labels


 [ 
https://issues.apache.org/jira/browse/HBASE-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-7663:
--

Attachment: (was: HBASE-7663.patch)

 [Per-KV security] Visibility labels
 ---

 Key: HBASE-7663
 URL: https://issues.apache.org/jira/browse/HBASE-7663
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, security
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Anoop Sam John

 Implement Accumulo-style visibility labels. Consider the following design 
 principles:
 - Coprocessor based implementation
 - Minimal to no changes to core code
 - Use KeyValue tags (HBASE-7448) to carry labels
 - Use OperationWithAttributes# {get,set}Attribute for handling visibility 
 labels in the API
 - Implement a new filter for evaluating visibility labels as KVs are streamed 
 through.
 This approach would be consistent in deployment and API details with other 
 per-KV security work, supporting environments where they might be both be 
 employed, even stacked on some tables.
 See the parent issue for more discussion.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-7663) [Per-KV security] Visibility labels


 [ 
https://issues.apache.org/jira/browse/HBASE-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-7663:
--

Attachment: HBASE-7663.patch

 [Per-KV security] Visibility labels
 ---

 Key: HBASE-7663
 URL: https://issues.apache.org/jira/browse/HBASE-7663
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, security
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Anoop Sam John
 Attachments: HBASE-7663.patch


 Implement Accumulo-style visibility labels. Consider the following design 
 principles:
 - Coprocessor based implementation
 - Minimal to no changes to core code
 - Use KeyValue tags (HBASE-7448) to carry labels
 - Use OperationWithAttributes# {get,set}Attribute for handling visibility 
 labels in the API
 - Implement a new filter for evaluating visibility labels as KVs are streamed 
 through.
 This approach would be consistent in deployment and API details with other 
 per-KV security work, supporting environments where they might be both be 
 employed, even stacked on some tables.
 See the parent issue for more discussion.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-7663) [Per-KV security] Visibility labels


[ 
https://issues.apache.org/jira/browse/HBASE-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797755#comment-13797755
 ] 

Anoop Sam John commented on HBASE-7663:
---

VisibilityController CP handles the visibility
The visibility labels are stored as tags with KVs
Use put.setCellVisibility(new CellVisibility(labelExp)); to add visibility 
expressions to cells
The label expression can contain visibility labels joined with logical 
expressions , | and !. Also using (, ) one can specify the precedence order
Eg : SECRET  CONFIDENTIAL  !PUBLIC

During read, (Scan/Get)  one can specify labels associated with that, in 
Authorizations
scan.setAuthorizations(new Authorizations(SECRET, CONFIDENTIAL));

 [Per-KV security] Visibility labels
 ---

 Key: HBASE-7663
 URL: https://issues.apache.org/jira/browse/HBASE-7663
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, security
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Anoop Sam John
 Attachments: HBASE-7663.patch


 Implement Accumulo-style visibility labels. Consider the following design 
 principles:
 - Coprocessor based implementation
 - Minimal to no changes to core code
 - Use KeyValue tags (HBASE-7448) to carry labels
 - Use OperationWithAttributes# {get,set}Attribute for handling visibility 
 labels in the API
 - Implement a new filter for evaluating visibility labels as KVs are streamed 
 through.
 This approach would be consistent in deployment and API details with other 
 per-KV security work, supporting environments where they might be both be 
 employed, even stacked on some tables.
 See the parent issue for more discussion.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-7663) [Per-KV security] Visibility labels


[ 
https://issues.apache.org/jira/browse/HBASE-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797759#comment-13797759
 ] 

Anoop Sam John commented on HBASE-7663:
---

https://reviews.apache.org/r/14709/

 [Per-KV security] Visibility labels
 ---

 Key: HBASE-7663
 URL: https://issues.apache.org/jira/browse/HBASE-7663
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, security
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Anoop Sam John
 Attachments: HBASE-7663.patch


 Implement Accumulo-style visibility labels. Consider the following design 
 principles:
 - Coprocessor based implementation
 - Minimal to no changes to core code
 - Use KeyValue tags (HBASE-7448) to carry labels
 - Use OperationWithAttributes# {get,set}Attribute for handling visibility 
 labels in the API
 - Implement a new filter for evaluating visibility labels as KVs are streamed 
 through.
 This approach would be consistent in deployment and API details with other 
 per-KV security work, supporting environments where they might be both be 
 employed, even stacked on some tables.
 See the parent issue for more discussion.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9689) Support using OperationAttributes through shell script


[ 
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797773#comment-13797773
 ] 

Hadoop QA commented on HBASE-9689:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12608896/HBASE-9689_3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7576//console

This message is automatically generated.

 Support using OperationAttributes through shell script
 --

 Key: HBASE-9689
 URL: https://issues.apache.org/jira/browse/HBASE-9689
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-9689_1.patch, HBASE-9689_2.patch, 
 HBASE-9689_3.patch, HBASE-9689.patch


 Currently the ruby scripts for Put does not support setting of Operation 
 Attributes through shell.  
 It may be useful in some cases and also for testing.  And that would give a 
 full  fledged support using shell.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HBASE-9790) If MAX_FILESIZE is negative, cluster will split regions indefinitely

2013-10-17 Thread Andrey Stepachev (JIRA)

Andrey Stepachev created HBASE-9790:
---

 Summary: If MAX_FILESIZE is negative, cluster will split regions 
indefinitely
 Key: HBASE-9790
 URL: https://issues.apache.org/jira/browse/HBASE-9790
 Project: HBase
  Issue Type: Bug
 Environment: 0.92.1-cdh4.1.1, Ubuntu 12.04
Reporter: Andrey Stepachev
Priority: Minor


'Alter' was issued with wrong (negative) max_file_size. At some moment 
regionservers begin to split regions producing zero size regions. Table won't 
to delete, event can't disable it.

Solution: stop cluster, delete table from hdfs, use with care  
{{hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair}}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9790) If MAX_FILESIZE is negative, cluster will split regions indefinitely

2013-10-17 Thread Andrey Stepachev (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Stepachev updated HBASE-9790:


Description: 
'Alter' was issued with wrong (negative) max_file_size. At some moment 
regionservers begin to split regions producing zero size regions. Table won't 
to delete, even can't disable it.

Solution: stop cluster, delete table from hdfs, use with care  
{{hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair}}

  was:
'Alter' was issued with wrong (negative) max_file_size. At some moment 
regionservers begin to split regions producing zero size regions. Table won't 
to delete, event can't disable it.

Solution: stop cluster, delete table from hdfs, use with care  
{{hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair}}


 If MAX_FILESIZE is negative, cluster will split regions indefinitely
 

 Key: HBASE-9790
 URL: https://issues.apache.org/jira/browse/HBASE-9790
 Project: HBase
  Issue Type: Bug
 Environment: 0.92.1-cdh4.1.1, Ubuntu 12.04
Reporter: Andrey Stepachev
Priority: Minor

 'Alter' was issued with wrong (negative) max_file_size. At some moment 
 regionservers begin to split regions producing zero size regions. Table won't 
 to delete, even can't disable it.
 Solution: stop cluster, delete table from hdfs, use with care  
 {{hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair}}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9720) TestSplitTransactionOnCluster#testShutdownFixupWhenDaughterHasSplit occasionally times out


[ 
https://issues.apache.org/jira/browse/HBASE-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797852#comment-13797852
 ] 

Hudson commented on HBASE-9720:
---

FAILURE: Integrated in hbase-0.96-hadoop2 #91 (See 
[https://builds.apache.org/job/hbase-0.96-hadoop2/91/])
HBASE-9720 TestSplitTransactionOnCluster#testShutdownFixupWhenDaughterHasSplit 
occasionally times out (tedyu: rev 1532962)
* 
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java


 TestSplitTransactionOnCluster#testShutdownFixupWhenDaughterHasSplit 
 occasionally times out
 --

 Key: HBASE-9720
 URL: https://issues.apache.org/jira/browse/HBASE-9720
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
 Fix For: 0.98.0, 0.96.1

 Attachments: 9720-v1.txt


 From 
 https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/779/testReport/junit/org.apache.hadoop.hbase.regionserver/TestSplitTransactionOnCluster/testShutdownFixupWhenDaughterHasSplit/
  :
 {code}
 java.lang.AssertionError: Waited too long for split
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at org.junit.Assert.assertFalse(Assert.java:64)
   at 
 org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.split(TestSplitTransactionOnCluster.java:1065)
   at 
 org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testShutdownFixupWhenDaughterHasSplit(TestSplitTransactionOnCluster.java:442)
 ...
 2013-10-05 13:00:18,060 DEBUG 
 [RS:2;quirinus:46584-smallCompactions-1380978003766] 
 regionserver.HRegionFileSystem(338): Committing store file 
 hdfs://localhost:45166/user/jenkins/hbase/data/default/testShutdownFixupWhenDaughterHasSplit/0d7218d1ce3bd629779009821908a3ed/.tmp/8b155b635b304a368e11dbd675d09312
  as 
 hdfs://localhost:45166/user/jenkins/hbase/data/default/testShutdownFixupWhenDaughterHasSplit/0d7218d1ce3bd629779009821908a3ed/info/8b155b635b304a368e11dbd675d09312
 2013-10-05 13:00:18,436 DEBUG [pool-1-thread-1-EventThread] 
 zookeeper.ZooKeeperWatcher(310): master:48355-0x14188b3d794 Received 
 ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, 
 path=/hbase/balancer
 2013-10-05 13:00:18,436 DEBUG [AM.ZK.Worker-pool2-t11] 
 master.AssignmentManager(818): Handling RS_ZK_REGION_SPLITTING, 
 server=quirinus.apache.org,46584,1380977990795, 
 region=e29b00c3bdaa3e10f6c4fe252a82399f, 
 current_state={e29b00c3bdaa3e10f6c4fe252a82399f state=SPLITTING, 
 ts=1380978012468, server=quirinus.apache.org,46584,1380977990795}
 {code}
 We can see the following stack traces at the end of test output:
 {code}
 Potentially hanging thread: RS:2;quirinus:46584-smallCompactions-1380978003766
   java.lang.Object.wait(Native Method)
   java.lang.Object.wait(Object.java:485)
   org.apache.hadoop.ipc.Client.call(Client.java:1333)
   org.apache.hadoop.ipc.Client.call(Client.java:1300)
   
 org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
   com.sun.proxy.$Proxy17.rename(Unknown Source)
   sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
   
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   java.lang.reflect.Method.invoke(Method.java:597)
   
 org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:188)
   
 org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
   com.sun.proxy.$Proxy17.rename(Unknown Source)
   
 org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.rename(ClientNamenodeProtocolTranslatorPB.java:396)
   sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   java.lang.reflect.Method.invoke(Method.java:597)
   org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:266)
   com.sun.proxy.$Proxy22.rename(Unknown Source)
   org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:1512)
   
 org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:528)
   org.apache.hadoop.fs.FilterFileSystem.rename(FilterFileSystem.java:210)
   
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.rename(HRegionFileSystem.java:924)
   
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:340)
   
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:312)
   
 org.apache.hadoop.hbase.regionserver.HStore.moveFileIntoPlace(HStore.java:1032)

[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code


 [ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-9754:
--

Attachment: 9754-rp-hregion-v2.txt

Good point, Chunhui.

In latest patch, I refactored Compactor#createFileScanners() and 
Compactor#setSmallestReadPoint() which are called by DefaultCompactor#compact().

Now store.getSmallestReadPoint() is only called once.

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9790) If MAX_FILESIZE is negative, cluster will split regions indefinitely

2013-10-17 Thread Jean-Marc Spaggiari (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797972#comment-13797972
 ] 

Jean-Marc Spaggiari commented on HBASE-9790:


Have you tried that on a more recent HBase version? Like with 0.94.x or CDH4.4 
if you use CDH?

 If MAX_FILESIZE is negative, cluster will split regions indefinitely
 

 Key: HBASE-9790
 URL: https://issues.apache.org/jira/browse/HBASE-9790
 Project: HBase
  Issue Type: Bug
 Environment: 0.92.1-cdh4.1.1, Ubuntu 12.04
Reporter: Andrey Stepachev
Priority: Minor

 'Alter' was issued with wrong (negative) max_file_size. At some moment 
 regionservers begin to split regions producing zero size regions. Table won't 
 to delete, even can't disable it.
 Solution: stop cluster, delete table from hdfs, use with care  
 {{hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair}}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online


[ 
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798031#comment-13798031
 ] 

Jimmy Xiang commented on HBASE-9773:


I will enhance the addendum a little to have a timeout, just like in the assign 
case.  I think we should not use runnable because the open will be retried 
usually.  If the unassign is timed out, the pool won't be blocked.

 Master aborted when hbck asked the master to assign a region that was already 
 online
 

 Key: HBASE-9773
 URL: https://issues.apache.org/jira/browse/HBASE-9773
 Project: HBase
  Issue Type: Bug
Reporter: Devaraj Das
Assignee: Jimmy Xiang
 Fix For: 0.98.0, 0.96.1

 Attachments: trunk-9773.addendum, trunk-9773.patch, 
 trunk-9773_v2.patch


 Came across this situation (with a version of 0.96 very close to RC5 version 
 created on 10/11):
 The sequence of events that happened:
 1. The hbck tool couldn't communicate with the RegionServer hosting namespace 
 region due to some security exceptions. hbck INCORRECTLY assumed the region 
 was not deployed.
 In output.log (client side):
 {noformat}
 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta = 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs = 
 hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
  deployed =  } not deployed on any region server.
 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
 {noformat}
 2. This led to the hbck tool trying to tell the master to assign the region.
 In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
 {noformat}
 2013-10-12 10:52:35,960 INFO  [RpcServer.handler=4,port=6] 
 master.HMaster: Client=hbase//172.18.145.105 assign 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
 {noformat}
 3. The master went through the steps - sent a CLOSE to the RegionServer 
 hosting namespace region.
 From master log:
 {noformat}
 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6] 
 master.AssignmentManager: Sent CLOSE to 
 gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for 
 region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
 {noformat}
 4. The master then tried to assign the namespace region to a region server, 
 and in the process ABORTED:
 From master log:
 {noformat}
 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6] 
 master.AssignmentManager: No previous transition plan found (or ignoring an 
 existing plan) for 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated 
 random 
 plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., 
 src=, 
 dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807; 
 4 (online=4, available=4) available servers, forceNewPlan=true
 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6] 
 master.HMaster: Master server abort: loaded coprocessors are: 
 [org.apache.hadoop.hbase.security.access.AccessController]
 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6] 
 master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a 
 state=OPEN, ts=1381564451344, 
 server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
  .. Cannot transit it to OFFLINE.
 java.lang.IllegalStateException: Unexpected state : 
 {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344, 
 server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
  .. Cannot transit it to OFFLINE.
 {noformat}
 {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK, 
 boolean forceNewPlan){code} is the method that does all the above. This was 
 called from the HMaster with true for both the boolean arguments.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HBASE-9791) MR initializes scanner twice

Jimmy Xiang created HBASE-9791:
--

 Summary: MR initializes scanner twice
 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


The first is in TableInputFormatBase.createRecordReader(). The second time is 
in initializing it.  We should not call initialize in creating the record 
reader.

{noformat}
2013-10-16 16:58:27,163 INFO [main] 
org.apache.hadoop.hbase.client.ScannerCallable: Open scanner=466730774138784884 
for 
scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 on region 
region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
 hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
ip:e1521.halxg.cloudera.com:36020
2013-10-16 16:58:27,164 INFO [main] 
org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
at 
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
at 
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
at 
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
at 
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
at 
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
at 
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
at 
org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
at 
org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
at 
org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
at 
org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
at 
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
2013-10-16 16:58:27,167 INFO [main] 
org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 
0 kvi 268435452(1073741808)
2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
mapreduce.task.io.sort.mb: 1024
2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
limit at 858993472
2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart 
= 0; bufvoid = 1073741824
2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 
268435452; length = 67108864
2013-10-16 16:58:27,903 INFO [main] 
org.apache.hadoop.hbase.client.ScannerCallable: Open 
scanner=7462140737850801183 for 
scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 on region 
region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
 hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
ip:e1521.halxg.cloudera.com:36020
2013-10-16 16:58:27,903 INFO [main] 
org.apache.hadoop.hbase.client.ScannerCallable: 
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
at 
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
at

[jira] [Updated] (HBASE-9791) MR initializes scanner twice


 [ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-9791:
---

Attachment: trunk-9791.patch

 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
   at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
 2013-10-16 16:58:27,167 INFO [main] 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
 output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
 2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: 
 (EQUATOR) 0 kvi 268435452(1073741808)
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 mapreduce.task.io.sort.mb: 1024
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
 limit at 858993472
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 bufstart = 0; bufvoid = 1073741824
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart 
 = 268435452; length = 67108864
 2013-10-16 16:58:27,903 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=7462140737850801183 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,903 INFO [main]

[jira] [Updated] (HBASE-9791) MR initializes scanner twice


 [ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-9791:
---

Status: Patch Available  (was: Open)

Added patch that removed the initialization during the creation time.  Tried 
TestImportExport which is still green.

 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
   at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
 2013-10-16 16:58:27,167 INFO [main] 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
 output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
 2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: 
 (EQUATOR) 0 kvi 268435452(1073741808)
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 mapreduce.task.io.sort.mb: 1024
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
 limit at 858993472
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 bufstart = 0; bufvoid = 1073741824
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart 
 = 268435452; length = 67108864
 2013-10-16 16:58:27,903 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=7462140737850801183 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098,

[jira] [Commented] (HBASE-9791) MR initializes scanner twice


[ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798048#comment-13798048
 ] 

stack commented on HBASE-9791:
--

+1

 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
   at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
 2013-10-16 16:58:27,167 INFO [main] 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
 output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
 2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: 
 (EQUATOR) 0 kvi 268435452(1073741808)
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 mapreduce.task.io.sort.mb: 1024
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
 limit at 858993472
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 bufstart = 0; bufvoid = 1073741824
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart 
 = 268435452; length = 67108864
 2013-10-16 16:58:27,903 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=7462140737850801183 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,903 INFO [main]

[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code


[ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798055#comment-13798055
 ] 

Hadoop QA commented on HBASE-9754:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12608945/9754-rp-hregion-v2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 27 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7577//console

This message is automatically generated.

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9689) Support using OperationAttributes through shell script


[ 
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798057#comment-13798057
 ] 

stack commented on HBASE-9689:
--

I skimmed the patch.  Looks good.  Thanks for going back and making it so we 
don't break BC.

 Support using OperationAttributes through shell script
 --

 Key: HBASE-9689
 URL: https://issues.apache.org/jira/browse/HBASE-9689
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-9689_1.patch, HBASE-9689_2.patch, 
 HBASE-9689_3.patch, HBASE-9689.patch


 Currently the ruby scripts for Put does not support setting of Operation 
 Attributes through shell.  
 It may be useful in some cases and also for testing.  And that would give a 
 full  fledged support using shell.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9689) Support using OperationAttributes through shell script


[ 
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798059#comment-13798059
 ] 

Andrew Purtell commented on HBASE-9689:
---

+1 on latest patch, thanks Ram

 Support using OperationAttributes through shell script
 --

 Key: HBASE-9689
 URL: https://issues.apache.org/jira/browse/HBASE-9689
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-9689_1.patch, HBASE-9689_2.patch, 
 HBASE-9689_3.patch, HBASE-9689.patch


 Currently the ruby scripts for Put does not support setting of Operation 
 Attributes through shell.  
 It may be useful in some cases and also for testing.  And that would give a 
 full  fledged support using shell.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HBASE-9792) Region states should update last assignments when a region is opened.

Jimmy Xiang created HBASE-9792:
--

 Summary: Region states should update last assignments when a 
region is opened.
 Key: HBASE-9792
 URL: https://issues.apache.org/jira/browse/HBASE-9792
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


Currently, we update a region's last assignment region server when the region 
is online.  We should do this sooner, when the region is moved to OPEN state.  
CM could kill this region server before we delete the znode and online the 
region.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (HBASE-7661) [Per-KV security] Store and apply per cell ACLs in a shadow CF


 [ 
https://issues.apache.org/jira/browse/HBASE-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-7661.
---

Resolution: Won't Fix

Now that we have tags and HFile V3, it's not necessary to use out of line 
storage for cell ACLs. Resolving as wontfix.

 [Per-KV security] Store and apply per cell ACLs in a shadow CF
 --

 Key: HBASE-7661
 URL: https://issues.apache.org/jira/browse/HBASE-7661
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, security
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Attachments: 7661-0.94.patch, 7661.patch


 Coprocessor based implementation of per-KV security that extends the existing 
 AccessController and ACL model to cover per cell permissions. More comments 
 on this approach can be found on the parent issue. Stores and consults the 
 additional metadata in a shadow column family managed by the 
 AccessController. Preserves existing user facing semantics. Does not require 
 any changes to core code except, optionally, the security shell commands, for 
 testing and prototyping convenience.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (HBASE-7664) [Per-KV security] Shell support


 [ 
https://issues.apache.org/jira/browse/HBASE-7664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-7664.
---

Resolution: Duplicate
  Assignee: (was: Andrew Purtell)

Effectively a dup of HBASE-9689

 [Per-KV security] Shell support
 ---

 Key: HBASE-7664
 URL: https://issues.apache.org/jira/browse/HBASE-7664
 Project: HBase
  Issue Type: Sub-task
  Components: security, shell
Affects Versions: 0.98.0
Reporter: Andrew Purtell

 Support simple exploration and validation of per-KV ACLs via the shell. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code


 [ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-9754:
--

Attachment: 9754-rp-hregion-v3.txt

New patch that replaces 0 read point parameter value with proper value.

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt, 9754-rp-hregion-v3.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9792) Region states should update last assignments when a region is opened.


 [ 
https://issues.apache.org/jira/browse/HBASE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-9792:
---

Attachment: trunk-9792.patch

 Region states should update last assignments when a region is opened.
 -

 Key: HBASE-9792
 URL: https://issues.apache.org/jira/browse/HBASE-9792
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: trunk-9792.patch


 Currently, we update a region's last assignment region server when the region 
 is online.  We should do this sooner, when the region is moved to OPEN state. 
  CM could kill this region server before we delete the znode and online the 
 region.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9792) Region states should update last assignments when a region is opened.


 [ 
https://issues.apache.org/jira/browse/HBASE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-9792:
---

Status: Patch Available  (was: Open)

Attached patch moved the last assignment update from regionOnline() to 
updateRegionStates().

 Region states should update last assignments when a region is opened.
 -

 Key: HBASE-9792
 URL: https://issues.apache.org/jira/browse/HBASE-9792
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: trunk-9792.patch


 Currently, we update a region's last assignment region server when the region 
 is online.  We should do this sooner, when the region is moved to OPEN state. 
  CM could kill this region server before we delete the znode and online the 
 region.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code


[ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798086#comment-13798086
 ] 

stack commented on HBASE-9754:
--

What are the implications of this change?

-MultiVersionConsistencyControl.resetThreadReadPoint();
 Scan scan = new Scan(get);
 InternalScanner scanner = (InternalScanner) store.getScanner(scan,
-scan.getFamilyMap().get(store.getFamily().getName()));
+scan.getFamilyMap().get(store.getFamily().getName()), 0);

We no longer do reset and we pass 0 which means all is readable?  Would think 
we'd retain at least the first line?

Does defaulting 0 in all tests change what is being tested (or do you think the 
reset was setting readpoint to 0 across the board anyways)?

This should be final now?

+  private long readPt = 0;

Is this an odd 'public' method to have in HRegion?

+   /*
+* Returns readpoint considering given IsolationLevel
+*/
+   public long getReadpoint(IsolationLevel isolationLevel) {
+ if (isolationLevel == IsolationLevel.READ_UNCOMMITTED) {
+   // This scan can read even uncommitted transactions
+   return Long.MAX_VALUE;
+ }
+ return mvcc.memstoreReadPoint();
+   }

especially as given just above, we can do a getMVCC so can go 
'behind-the-scenes'.  Should it be private?

It is the readPoint for the region?

Remove

-MultiVersionConsistencyControl.setThreadReadPoint(this.readPt);
+// MultiVersionConsistencyControl.setThreadReadPoint(this.readPt);

ditto

-MultiVersionConsistencyControl.setThreadReadPoint(this.readPt);
+// MultiVersionConsistencyControl.setThreadReadPoint(this.readPt);

Undocumented passing of zero (explaining why this is ok) and the sometimes 
passing of smallestreadpoint instead of readpoint is unsettling.  Explaination 
on why it is ok in the former case and what smallest means in latter would help 
the folks who come after understand what is going on.

Otherwise patch is good.



 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt, 9754-rp-hregion-v3.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9767) Support OperationAttributes in ImportTSV Parser


[ 
https://issues.apache.org/jira/browse/HBASE-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798093#comment-13798093
 ] 

Andrew Purtell commented on HBASE-9767:
---

+1

 Support OperationAttributes in ImportTSV Parser
 ---

 Key: HBASE-9767
 URL: https://issues.apache.org/jira/browse/HBASE-9767
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Attachments: HBASE-9767.patch


 This JIRA aims at supporting the operation attributes in bulk loads.  Ideally 
 this operation attributes once extracted has to be set in the 
 mappers/reducers.
 In case of mappers using TableOutPutFormat this would be very helpful when 
 the puts are done through HTable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9689) Support using OperationAttributes through shell script


[ 
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798096#comment-13798096
 ] 

Anoop Sam John commented on HBASE-9689:
---

LGTM

 Support using OperationAttributes through shell script
 --

 Key: HBASE-9689
 URL: https://issues.apache.org/jira/browse/HBASE-9689
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-9689_1.patch, HBASE-9689_2.patch, 
 HBASE-9689_3.patch, HBASE-9689.patch


 Currently the ruby scripts for Put does not support setting of Operation 
 Attributes through shell.  
 It may be useful in some cases and also for testing.  And that would give a 
 full  fledged support using shell.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code


[ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798123#comment-13798123
 ] 

Ted Yu commented on HBASE-9754:
---

bq. What are the implications of this change?
You were referring to the change in HBaseTestingUtility.java, right ?

Here is what resetThreadReadPoint() does:

-  public static void resetThreadReadPoint() {
-perThreadReadPoint.set(0L);
-  }

Meaning, effective readpoint is reset to 0. Hence passing 0 to 
store.getScanner() is consistent with the existing test.

bq. Would think we'd retain at least the first line?
Is the first line MultiVersionConsistencyControl.resetThreadReadPoint() ? That 
method has been removed.

bq. Does defaulting 0 in all tests change what is being tested
There is no change in semantics. 0 is only used when the two StoreScanner 
ctors, marked for testing, are used.

bq. This should be final now?
Done.

bq. Is this an odd 'public' method to have in HRegion?
It is used by NoOpScanPolicyObserver. Compilation passes when I change it to 
package private. My reasoning was that if user's RegionObserver overrides 
preStoreScannerOpen(), this method is needed. User's code would not be in 
org.apache.hadoop.hbase.regionserver package.

bq. It is the readPoint for the region?
Readpoint is associated with mvcc for the region.

bq. Undocumented passing of zero (explaining why this is ok)
Will add doc in next patch.

bq. and what smallest means in latter
Would do this too.

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt, 9754-rp-hregion-v3.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible


 [ 
https://issues.apache.org/jira/browse/HBASE-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-9778:
-

Fix Version/s: (was: 0.96.1)
   (was: 0.94.13)
   (was: 0.98.0)

Unscheduling for now.

 Avoid seeking to next column in ExplicitColumnTracker when possible
 ---

 Key: HBASE-9778
 URL: https://issues.apache.org/jira/browse/HBASE-9778
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Attachments: 9778-0.94.txt, 9778-0.94-v2.txt, 9778-0.94-v3.txt, 
 9778-trunk.txt, 9778-trunk-v2.txt, 9778-trunk-v3.txt


 The issue of slow seeking in ExplicitColumnTracker was brought up by 
 [~vrodionov] on the dev list.
 My idea here is to avoid the seeking if we know that there aren't many 
 versions to skip.
 How do we know? We'll use the column family's VERSIONS setting as a hint. If 
 VERSIONS is set to 1 (or maybe some value  10) we'll avoid the seek and call 
 SKIP repeatedly.
 HBASE-9769 has some initial number for this approach:
 Interestingly it depends on which column(s) is (are) selected.
 Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, 
 everything filtered at the server with a ValueFilter. Everything measured in 
 seconds.
 Without patch:
 ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4||
 |6.4|8.5|14.3|14.6|11.1|20.3|
 With patch:
 ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4||
 |6.4|8.4|8.9|9.9|6.4|10.0|
 Variation here was +- 0.2s.
 So with this patch scanning is 2x faster than without in some cases, and 
 never slower. No special hint needed, beyond declaring VERSIONS correctly.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Comment Edited] (HBASE-9754) Consider eliminating threadlocals from MVCC code


[ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798123#comment-13798123
 ] 

Ted Yu edited comment on HBASE-9754 at 10/17/13 5:23 PM:
-

bq. What are the implications of this change?
You were referring to the change in HBaseTestingUtility.java, right ?

Here is what resetThreadReadPoint() does:
{code}
-  public static void resetThreadReadPoint() {
-perThreadReadPoint.set(0L);
-  }
{code}
Meaning, effective readpoint is reset to 0. Hence passing 0 to 
store.getScanner() is consistent with the existing test.

bq. Would think we'd retain at least the first line?
Is the first line MultiVersionConsistencyControl.resetThreadReadPoint() ? That 
method has been removed.

bq. Does defaulting 0 in all tests change what is being tested
There is no change in semantics. 0 is only used when the two StoreScanner 
ctors, marked for testing, are used.

bq. This should be final now?
Done.

bq. Is this an odd 'public' method to have in HRegion?
It is used by NoOpScanPolicyObserver. Compilation passes when I change it to 
package private. My reasoning was that if user's RegionObserver overrides 
preStoreScannerOpen(), this method is needed. User's code would not be in 
org.apache.hadoop.hbase.regionserver package.

bq. It is the readPoint for the region?
Readpoint is associated with mvcc for the region.

bq. Undocumented passing of zero (explaining why this is ok)
Will add doc in next patch.

bq. and what smallest means in latter
Would do this too.


was (Author: yuzhih...@gmail.com):
bq. What are the implications of this change?
You were referring to the change in HBaseTestingUtility.java, right ?

Here is what resetThreadReadPoint() does:

-  public static void resetThreadReadPoint() {
-perThreadReadPoint.set(0L);
-  }

Meaning, effective readpoint is reset to 0. Hence passing 0 to 
store.getScanner() is consistent with the existing test.

bq. Would think we'd retain at least the first line?
Is the first line MultiVersionConsistencyControl.resetThreadReadPoint() ? That 
method has been removed.

bq. Does defaulting 0 in all tests change what is being tested
There is no change in semantics. 0 is only used when the two StoreScanner 
ctors, marked for testing, are used.

bq. This should be final now?
Done.

bq. Is this an odd 'public' method to have in HRegion?
It is used by NoOpScanPolicyObserver. Compilation passes when I change it to 
package private. My reasoning was that if user's RegionObserver overrides 
preStoreScannerOpen(), this method is needed. User's code would not be in 
org.apache.hadoop.hbase.regionserver package.

bq. It is the readPoint for the region?
Readpoint is associated with mvcc for the region.

bq. Undocumented passing of zero (explaining why this is ok)
Will add doc in next patch.

bq. and what smallest means in latter
Would do this too.

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt, 9754-rp-hregion-v3.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HBASE-9793) Offline a region before it's closed could cause double assignment

Jimmy Xiang created HBASE-9793:
--

 Summary: Offline a region before it's closed could cause double 
assignment
 Key: HBASE-9793
 URL: https://issues.apache.org/jira/browse/HBASE-9793
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


The fix for HBASE-9773 could cause double assignment, as [~jeffreyz] pointed 
out. Let's fix it in a separate jira instead of an addendum since there are 
different opinions on how to fix it.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible


[ 
https://issues.apache.org/jira/browse/HBASE-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798131#comment-13798131
 ] 

Lars Hofhansl commented on HBASE-9778:
--

I'd like to do some brainstorming about how we can make reseek faster if the 
reseek distance is expected to be small. Currently the absolute worst case is 
when reseek just seeks to the next KV, where next is orders of magnitudes 
faster.

Unfortunately we hit this worst case all the time if there is only one version 
of each KV.

 Avoid seeking to next column in ExplicitColumnTracker when possible
 ---

 Key: HBASE-9778
 URL: https://issues.apache.org/jira/browse/HBASE-9778
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Attachments: 9778-0.94.txt, 9778-0.94-v2.txt, 9778-0.94-v3.txt, 
 9778-trunk.txt, 9778-trunk-v2.txt, 9778-trunk-v3.txt


 The issue of slow seeking in ExplicitColumnTracker was brought up by 
 [~vrodionov] on the dev list.
 My idea here is to avoid the seeking if we know that there aren't many 
 versions to skip.
 How do we know? We'll use the column family's VERSIONS setting as a hint. If 
 VERSIONS is set to 1 (or maybe some value  10) we'll avoid the seek and call 
 SKIP repeatedly.
 HBASE-9769 has some initial number for this approach:
 Interestingly it depends on which column(s) is (are) selected.
 Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, 
 everything filtered at the server with a ValueFilter. Everything measured in 
 seconds.
 Without patch:
 ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4||
 |6.4|8.5|14.3|14.6|11.1|20.3|
 With patch:
 ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4||
 |6.4|8.4|8.9|9.9|6.4|10.0|
 Variation here was +- 0.2s.
 So with this patch scanning is 2x faster than without in some cases, and 
 never slower. No special hint needed, beyond declaring VERSIONS correctly.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online


 [ 
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang resolved HBASE-9773.


Resolution: Fixed

[~jeffreyz], let's close this one and fix the double assignment issue in 
HBASE-9793.  It may need some discussions.

 Master aborted when hbck asked the master to assign a region that was already 
 online
 

 Key: HBASE-9773
 URL: https://issues.apache.org/jira/browse/HBASE-9773
 Project: HBase
  Issue Type: Bug
Reporter: Devaraj Das
Assignee: Jimmy Xiang
 Fix For: 0.98.0, 0.96.1

 Attachments: trunk-9773.addendum, trunk-9773.patch, 
 trunk-9773_v2.patch


 Came across this situation (with a version of 0.96 very close to RC5 version 
 created on 10/11):
 The sequence of events that happened:
 1. The hbck tool couldn't communicate with the RegionServer hosting namespace 
 region due to some security exceptions. hbck INCORRECTLY assumed the region 
 was not deployed.
 In output.log (client side):
 {noformat}
 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta = 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs = 
 hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
  deployed =  } not deployed on any region server.
 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
 {noformat}
 2. This led to the hbck tool trying to tell the master to assign the region.
 In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
 {noformat}
 2013-10-12 10:52:35,960 INFO  [RpcServer.handler=4,port=6] 
 master.HMaster: Client=hbase//172.18.145.105 assign 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
 {noformat}
 3. The master went through the steps - sent a CLOSE to the RegionServer 
 hosting namespace region.
 From master log:
 {noformat}
 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6] 
 master.AssignmentManager: Sent CLOSE to 
 gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for 
 region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
 {noformat}
 4. The master then tried to assign the namespace region to a region server, 
 and in the process ABORTED:
 From master log:
 {noformat}
 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6] 
 master.AssignmentManager: No previous transition plan found (or ignoring an 
 existing plan) for 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated 
 random 
 plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., 
 src=, 
 dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807; 
 4 (online=4, available=4) available servers, forceNewPlan=true
 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6] 
 master.HMaster: Master server abort: loaded coprocessors are: 
 [org.apache.hadoop.hbase.security.access.AccessController]
 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6] 
 master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a 
 state=OPEN, ts=1381564451344, 
 server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
  .. Cannot transit it to OFFLINE.
 java.lang.IllegalStateException: Unexpected state : 
 {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344, 
 server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
  .. Cannot transit it to OFFLINE.
 {noformat}
 {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK, 
 boolean forceNewPlan){code} is the method that does all the above. This was 
 called from the HMaster with true for both the boolean arguments.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9689) Support using OperationAttributes through shell script


 [ 
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-9689:
--

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to trunk.  Thanks Andy, Anoop and Stack for the reviews.

 Support using OperationAttributes through shell script
 --

 Key: HBASE-9689
 URL: https://issues.apache.org/jira/browse/HBASE-9689
 Project: HBase
  Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.98.0

 Attachments: HBASE-9689_1.patch, HBASE-9689_2.patch, 
 HBASE-9689_3.patch, HBASE-9689.patch


 Currently the ruby scripts for Put does not support setting of Operation 
 Attributes through shell.  
 It may be useful in some cases and also for testing.  And that would give a 
 full  fledged support using shell.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9789) Change logging for Coprocessor exec call to trace


[ 
https://issues.apache.org/jira/browse/HBASE-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798144#comment-13798144
 ] 

Lars Hofhansl commented on HBASE-9789:
--

Actually 0.96+ does not have this. Coprocs are protobug based now and I do not 
see each invocation logged.

 Change logging for Coprocessor exec call to trace
 -

 Key: HBASE-9789
 URL: https://issues.apache.org/jira/browse/HBASE-9789
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.13


 Currently both RegionServer and HMaster log (debug) this
 Received dynamic protocol exec call with protocolName class
 on each coprocessor exec call.
 We just filled our RegionServer log with 160gb of these since we're making 
 heavy use of coprocessors.
 I would like to change this to trace. Any objections?



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9789) Change logging for Coprocessor exec call to trace


 [ 
https://issues.apache.org/jira/browse/HBASE-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-9789:
-

Fix Version/s: (was: 0.96.1)
   (was: 0.98.0)

 Change logging for Coprocessor exec call to trace
 -

 Key: HBASE-9789
 URL: https://issues.apache.org/jira/browse/HBASE-9789
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.13


 Currently both RegionServer and HMaster log (debug) this
 Received dynamic protocol exec call with protocolName class
 on each coprocessor exec call.
 We just filled our RegionServer log with 160gb of these since we're making 
 heavy use of coprocessors.
 I would like to change this to trace. Any objections?



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)

2013-10-17 Thread Himanshu Vashishtha (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798145#comment-13798145
 ] 

Himanshu Vashishtha commented on HBASE-8741:


Filed HBASE-9785 for TestHeapSize test failure.

 Scope sequenceid to the region rather than regionserver (WAS: Mutations on 
 Regions in recovery mode might have same sequenceIDs)
 

 Key: HBASE-8741
 URL: https://issues.apache.org/jira/browse/HBASE-8741
 Project: HBase
  Issue Type: Bug
  Components: MTTR
Affects Versions: 0.95.1
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
 Fix For: 0.98.0

 Attachments: HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, 
 HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, 
 HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, 
 HBASE-8741-v5.patch


 Currently, when opening a region, we find the maximum sequence ID from all 
 its HFiles and then set the LogSequenceId of the log (in case the later is at 
 a small value). This works good in recovered.edits case as we are not writing 
 to the region until we have replayed all of its previous edits. 
 With distributed log replay, if we want to enable writes while a region is 
 under recovery, we need to make sure that the logSequenceId  maximum 
 logSequenceId of the old regionserver. Otherwise, we might have a situation 
 where new edits have same (or smaller) sequenceIds. 
 We can store region level information in the WALTrailer, than this scenario 
 could be avoided by:
 a) reading the trailer of the last completed file, i.e., last wal file 
 which has a trailer and,
 b) completely reading the last wal file (this file would not have the 
 trailer, so it needs to be read completely).
 In future, if we switch to multi wal file, we could read the trailer for all 
 completed WAL files, and reading the remaining incomplete files.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9789) Change logging for Coprocessor exec call to trace


 [ 
https://issues.apache.org/jira/browse/HBASE-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-9789:
-

Attachment: 9789.txt

Simple patch. Will commit soon (0.94 only), unless I hear objections.

 Change logging for Coprocessor exec call to trace
 -

 Key: HBASE-9789
 URL: https://issues.apache.org/jira/browse/HBASE-9789
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.13

 Attachments: 9789.txt


 Currently both RegionServer and HMaster log (debug) this
 Received dynamic protocol exec call with protocolName class
 on each coprocessor exec call.
 We just filled our RegionServer log with 160gb of these since we're making 
 heavy use of coprocessors.
 I would like to change this to trace. Any objections?



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Comment Edited] (HBASE-9789) Change logging for Coprocessor exec call to trace


[ 
https://issues.apache.org/jira/browse/HBASE-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798144#comment-13798144
 ] 

Lars Hofhansl edited comment on HBASE-9789 at 10/17/13 5:41 PM:


Actually 0.96+ does not have this. Coprocs are protobuf based now and I do not 
see each invocation logged.


was (Author: lhofhansl):
Actually 0.96+ does not have this. Coprocs are protobug based now and I do not 
see each invocation logged.

 Change logging for Coprocessor exec call to trace
 -

 Key: HBASE-9789
 URL: https://issues.apache.org/jira/browse/HBASE-9789
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.13

 Attachments: 9789.txt


 Currently both RegionServer and HMaster log (debug) this
 Received dynamic protocol exec call with protocolName class
 on each coprocessor exec call.
 We just filled our RegionServer log with 160gb of these since we're making 
 heavy use of coprocessors.
 I would like to change this to trace. Any objections?



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9749) Custom threadpool for Coprocessor obtained HTables

2013-10-17 Thread Jesse Yates (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798161#comment-13798161
 ] 

Jesse Yates commented on HBASE-9749:


What are you getting at [~saint@gmail.com]? External interface and 
functionality to users is the same, this way just minimizes the possible code 
paths.

 Custom threadpool for Coprocessor obtained HTables
 --

 Key: HBASE-9749
 URL: https://issues.apache.org/jira/browse/HBASE-9749
 Project: HBase
  Issue Type: Improvement
  Components: Coprocessors
Affects Versions: 0.98.0, 0.94.13, 0.96.1
Reporter: Jesse Yates
Assignee: Jesse Yates
 Attachments: hbase-9749-0.94-v0.patch, hbase-9749-0.96-v0.patch, 
 hbase-9749-trunk-v0.patch


 Coprocessors currently can only use the default HTable constructor that takes 
 a single thread-pool.This is overly constrictive for coprocessors that desire 
 tighter control over their resources.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online


[ 
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798164#comment-13798164
 ] 

Jeffrey Zhong commented on HBASE-9773:
--

{quote}
I will enhance the addendum a little to have a timeout, just like in the assign 
case. I think we should not use runnable because the open will be retried 
usually. If the unassign is timed out, the pool won't be blocked.
{quote}
Since TimeoutMonitor is disable by default so the open won't retried. While for 
this JIRA issue, if we timeout unassign is a reasonable solution because in SSH 
case the host server is dead already so unassign will bail out quick without 
timeout. For other situations, we give up assignment is acceptable because 
region assignment is more like a hint to the system and master doesn't have to 
assign a region by user requests. 

 Master aborted when hbck asked the master to assign a region that was already 
 online
 

 Key: HBASE-9773
 URL: https://issues.apache.org/jira/browse/HBASE-9773
 Project: HBase
  Issue Type: Bug
Reporter: Devaraj Das
Assignee: Jimmy Xiang
 Fix For: 0.98.0, 0.96.1

 Attachments: trunk-9773.addendum, trunk-9773.patch, 
 trunk-9773_v2.patch


 Came across this situation (with a version of 0.96 very close to RC5 version 
 created on 10/11):
 The sequence of events that happened:
 1. The hbck tool couldn't communicate with the RegionServer hosting namespace 
 region due to some security exceptions. hbck INCORRECTLY assumed the region 
 was not deployed.
 In output.log (client side):
 {noformat}
 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta = 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs = 
 hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
  deployed =  } not deployed on any region server.
 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
 {noformat}
 2. This led to the hbck tool trying to tell the master to assign the region.
 In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
 {noformat}
 2013-10-12 10:52:35,960 INFO  [RpcServer.handler=4,port=6] 
 master.HMaster: Client=hbase//172.18.145.105 assign 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
 {noformat}
 3. The master went through the steps - sent a CLOSE to the RegionServer 
 hosting namespace region.
 From master log:
 {noformat}
 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6] 
 master.AssignmentManager: Sent CLOSE to 
 gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for 
 region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
 {noformat}
 4. The master then tried to assign the namespace region to a region server, 
 and in the process ABORTED:
 From master log:
 {noformat}
 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6] 
 master.AssignmentManager: No previous transition plan found (or ignoring an 
 existing plan) for 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated 
 random 
 plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., 
 src=, 
 dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807; 
 4 (online=4, available=4) available servers, forceNewPlan=true
 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6] 
 master.HMaster: Master server abort: loaded coprocessors are: 
 [org.apache.hadoop.hbase.security.access.AccessController]
 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6] 
 master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a 
 state=OPEN, ts=1381564451344, 
 server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
  .. Cannot transit it to OFFLINE.
 java.lang.IllegalStateException: Unexpected state : 
 {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344, 
 server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
  .. Cannot transit it to OFFLINE.
 {noformat}
 {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK, 
 boolean forceNewPlan){code} is the method that does all the above. This was 
 called from the HMaster with true for both the boolean arguments.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-8912) [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE


[ 
https://issues.apache.org/jira/browse/HBASE-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798162#comment-13798162
 ] 

Lars Hofhansl commented on HBASE-8912:
--

Some folks on the mailing list had something like this happening to them.

 [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to 
 OFFLINE
 --

 Key: HBASE-8912
 URL: https://issues.apache.org/jira/browse/HBASE-8912
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
 Fix For: 0.94.13

 Attachments: HBase-0.94 #1036 test - testRetrying [Jenkins].html


 AM throws this exception which subsequently causes the master to abort: 
 {code}
 java.lang.IllegalStateException: Unexpected state : 
 testRetrying,jjj,1372891751115.9b828792311001062a5ff4b1038fe33b. 
 state=PENDING_OPEN, ts=1372891751912, 
 server=hemera.apache.org,39064,1372891746132 .. Cannot transit it to OFFLINE.
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
   at 
 org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
   at java.lang.Thread.run(Thread.java:662)
 {code}
 This exception trace is from the failing test TestMetaReaderEditor which is 
 failing pretty frequently, but looking at the test code, I think this is not 
 a test-only issue, but affects the main code path. 
 https://builds.apache.org/job/HBase-0.94/1036/testReport/junit/org.apache.hadoop.hbase.catalog/TestMetaReaderEditor/testRetrying/



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9793) Offline a region before it's closed could cause double assignment


 [ 
https://issues.apache.org/jira/browse/HBASE-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-9793:
---

Attachment: trunk-9793.patch

Attached a similar patch as the addendum for HBASE-9773.  Added a timeout 
similar to that one for assign() to avoid waiting for too long while the thread 
is blocked.

 Offline a region before it's closed could cause double assignment
 -

 Key: HBASE-9793
 URL: https://issues.apache.org/jira/browse/HBASE-9793
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: trunk-9793.patch


 The fix for HBASE-9773 could cause double assignment, as [~jeffreyz] pointed 
 out. Let's fix it in a separate jira instead of an addendum since there are 
 different opinions on how to fix it.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online


[ 
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798171#comment-13798171
 ] 

Jimmy Xiang commented on HBASE-9773:


[~jeffreyz], I posted a patch on HBASE-9793. Could you please take a look? 
Thanks.

 Master aborted when hbck asked the master to assign a region that was already 
 online
 

 Key: HBASE-9773
 URL: https://issues.apache.org/jira/browse/HBASE-9773
 Project: HBase
  Issue Type: Bug
Reporter: Devaraj Das
Assignee: Jimmy Xiang
 Fix For: 0.98.0, 0.96.1

 Attachments: trunk-9773.addendum, trunk-9773.patch, 
 trunk-9773_v2.patch


 Came across this situation (with a version of 0.96 very close to RC5 version 
 created on 10/11):
 The sequence of events that happened:
 1. The hbck tool couldn't communicate with the RegionServer hosting namespace 
 region due to some security exceptions. hbck INCORRECTLY assumed the region 
 was not deployed.
 In output.log (client side):
 {noformat}
 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta = 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs = 
 hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
  deployed =  } not deployed on any region server.
 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
 {noformat}
 2. This led to the hbck tool trying to tell the master to assign the region.
 In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
 {noformat}
 2013-10-12 10:52:35,960 INFO  [RpcServer.handler=4,port=6] 
 master.HMaster: Client=hbase//172.18.145.105 assign 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
 {noformat}
 3. The master went through the steps - sent a CLOSE to the RegionServer 
 hosting namespace region.
 From master log:
 {noformat}
 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6] 
 master.AssignmentManager: Sent CLOSE to 
 gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for 
 region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
 {noformat}
 4. The master then tried to assign the namespace region to a region server, 
 and in the process ABORTED:
 From master log:
 {noformat}
 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6] 
 master.AssignmentManager: No previous transition plan found (or ignoring an 
 existing plan) for 
 hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated 
 random 
 plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., 
 src=, 
 dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807; 
 4 (online=4, available=4) available servers, forceNewPlan=true
 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6] 
 master.HMaster: Master server abort: loaded coprocessors are: 
 [org.apache.hadoop.hbase.security.access.AccessController]
 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6] 
 master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a 
 state=OPEN, ts=1381564451344, 
 server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
  .. Cannot transit it to OFFLINE.
 java.lang.IllegalStateException: Unexpected state : 
 {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344, 
 server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
  .. Cannot transit it to OFFLINE.
 {noformat}
 {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK, 
 boolean forceNewPlan){code} is the method that does all the above. This was 
 called from the HMaster with true for both the boolean arguments.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code


 [ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-9754:
--

Attachment: (was: 9754-rp-hregion-v3.txt)

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HBASE-9794) KeyValues / cells backed by buffer fragments

Andrew Purtell created HBASE-9794:
-

 Summary: KeyValues / cells backed by buffer fragments
 Key: HBASE-9794
 URL: https://issues.apache.org/jira/browse/HBASE-9794
 Project: HBase
  Issue Type: Brainstorming
Reporter: Andrew Purtell


There are various places in the code where we see comments to the effect would 
be great if we had a scatter gather API for KV, appearing at places where we 
rewrite KVs on the server, for example in HRegion where we process appends and 
increments.

KeyValues are stored in buffers of fixed length. This approach has performance 
advantages for the common case where KVs are not manipulated on their way from 
disk to RPC. The disadvantage of this approach is any manipulation of tags 
requires the creation of a new buffer to hold the result, and a copy of the KV 
data into the new buffer. Appends and increments are typically a small 
percentage of overall workload so this has been fine up to now.
 
KeyValues can now carry metadata known as tags. Tags are stored contiguously 
with the rest of the KeyValue. Applications wishing to use tags (like per cell 
security) change the equation by wanting to rewrite KVs significantly more 
often. 

We should consider backing KeyValue with an alternative structure that can 
better support rewriting portions of its data, appends to existing buffers, 
scatter-gather copies, possibly even copy-on-write.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code


 [ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-9754:
--

Attachment: 9754-rp-hregion-v3.txt

Updated 9754-rp-hregion-v3.txt with comment on why 0 is passed as readpoint in 
certain places.

Also replaced smallestReadPoint in tests.

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (HBASE-9789) Change logging for Coprocessor exec call to trace


 [ 
https://issues.apache.org/jira/browse/HBASE-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl resolved HBASE-9789.
--

Resolution: Fixed

Committed to 0.94.

 Change logging for Coprocessor exec call to trace
 -

 Key: HBASE-9789
 URL: https://issues.apache.org/jira/browse/HBASE-9789
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.13

 Attachments: 9789.txt


 Currently both RegionServer and HMaster log (debug) this
 Received dynamic protocol exec call with protocolName class
 on each coprocessor exec call.
 We just filled our RegionServer log with 160gb of these since we're making 
 heavy use of coprocessors.
 I would like to change this to trace. Any objections?



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9747) PrefixFilter with OR condition gives wrong results


[ 
https://issues.apache.org/jira/browse/HBASE-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798179#comment-13798179
 ] 

Lars Hofhansl commented on HBASE-9747:
--

Actually, why is filterRow initialized to true (rather than false)? 
[~adityakishore]


 PrefixFilter with OR condition gives wrong results
 --

 Key: HBASE-9747
 URL: https://issues.apache.org/jira/browse/HBASE-9747
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 0.95.2, 0.94.9
Reporter: Deepa Remesh
Assignee: Aditya Kishore
 Attachments: HBASE-9747_0.94.patch, HBASE-9747.patch


 PrefixFilter when used with a SingleColumnValueFilter with an OR condition 
 gives wrong results. In below example, each filter when evaluated separately 
 gives 1 row each. The OR condition with the two filters gives 3 rows instead 
 of 2. Repro below:
 create 't1', 'f1'
 put 't1','a1','f1:q2','111'
 put 't1','b1','f1:q1','112'
 put 't1','c1','f1:q1','113'
 hbase(main):020:0 scan 't1', {FILTER = PrefixFilter ('b') OR 
 SingleColumnValueFilter('f1', 'q1', =, 'binary:113')}
 ROWCOLUMN+CELL
  a1column=f1:q2, 
 timestamp=1381468905492, value=111
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
  c1column=f1:q1, 
 timestamp=1381468905549, value=113
 3 row(s) in 0.1020 seconds
 hbase(main):021:0 scan 't1', {FILTER = PrefixFilter ('b')}
 ROWCOLUMN+CELL
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
 1 row(s) in 0.0150 seconds
 hbase(main):002:0 scan 't1', {FILTER = SingleColumnValueFilter('f1', 'q1', 
 =, 'binary:113')}
 ROWCOLUMN+CELL
  c1column=f1:q1, 
 timestamp=1381469178679, value=113
 1 row(s) in 0.0140 seconds



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9794) KeyValues / cells backed by buffer fragments

[
https://issues.apache.org/jira/browse/HBASE-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798187#comment-13798187
]

Lars Hofhansl commented on HBASE-9794:
--

Related to (or dup of) HBASE-7320?

KeyValues / cells backed by buffer fragments

Key: HBASE-9794
URL: https://issues.apache.org/jira/browse/HBASE-9794
Project: HBase
Issue Type: Brainstorming
Reporter: Andrew Purtell

There are various places in the code where we see comments to the effect
would be great if we had a scatter gather API for KV, appearing at places
where we rewrite KVs on the server, for example in HRegion where we process
appends and increments.
KeyValues are stored in buffers of fixed length. This approach has
performance advantages for the common case where KVs are not manipulated on
their way from disk to RPC. The disadvantage of this approach is any
manipulation of tags requires the creation of a new buffer to hold the
result, and a copy of the KV data into the new buffer. Appends and increments
are typically a small percentage of overall workload so this has been fine up
to now.

KeyValues can now carry metadata known as tags. Tags are stored contiguously
with the rest of the KeyValue. Applications wishing to use tags (like per
cell security) change the equation by wanting to rewrite KVs significantly
more often.
We should consider backing KeyValue with an alternative structure that can
better support rewriting portions of its data, appends to existing buffers,
scatter-gather copies, possibly even copy-on-write.

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9794) KeyValues / cells backed by buffer fragments


[ 
https://issues.apache.org/jira/browse/HBASE-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798190#comment-13798190
 ] 

Andrew Purtell commented on HBASE-9794:
---

Considering alternatives to the single contiguous buffer as we have now, but 
only if needed. The current way should remain the default way.

The kind of KeyValue manipulations desired here are analogous to those 
performed by network stacks in operating systems. 

The BSD mbuf structure is a good example, although it contains a lot of 
particulars to network stacks.

Back when the dinosaurs roamed the earth I worked with a research OS called 
Scout which had (I though) a particularly nice tree based structure for 
composing and decomposing packet buffers. Could be inspiration. From foggy 
memory it had an API like:

{code}
// Return the current length of the message
extern size_t msgLength (Msg m);

// Replace the contents of message 'm' with that of 'other', but without an 
ownership transfer to 'm', any changes will have COW semantics
extern void msgAssign (Msg m, Msg other);

// Replace the contents of message 'm' with the union of 'left' and 'write', 
but without an ownership transfer to 'm', any changes will have COW semantics
extern void msgJoin (Msg m, Msg left, Msg right);

// Remove 'len' bytes from the head of message 'm' into message 'other'
extern void msgBreak (Msg m, Msg other, size_t len);

// Get a contiguous view over 'len' bytes of new storage at the tail of message 
'm', may cause internal tree manipulations and allocations in order to provide 
it
extern void * msgPush (Msg m, size_t len);

// Get a contiguous view over 'len' bytes at the head of message 'm', may cause 
internal tree manipulations and allocations in order to provide it, and remove 
those bytes from the message
extern void * msgPop (Msg m, size_t len);

// Get a contiguous view over 'len' bytes at the head of message 'm', may cause 
internal tree manipulations and allocations in order to provide it, without 
removing those bytes from the message
extern void * msgPeek (Msg m, size_t len);

// Discard 'len' bytes from the head of the message
extern void msgDiscard (Msg m, size_t len);

// Discard 'len' bytes from the tail of the message
extern void msgTruncate (Msg m, size_t len);

// Initialize state for a walk over the tree of buffers for message 'm'
extern void msgWalkInit (MsgWalk w, Msg m);

// Return a view over the contents of the next buffer for message 'm', or the 
first buffer upon first invocation. Does not trigger any tree manipulations or 
allocations
extern void * msgWalkNext (MsgWalk w, size_t *lenp);

// Clean up walk state
extern void msgWalkDone (MsgWalk w);
{code}

 KeyValues / cells backed by buffer fragments
 

 Key: HBASE-9794
 URL: https://issues.apache.org/jira/browse/HBASE-9794
 Project: HBase
  Issue Type: Brainstorming
Reporter: Andrew Purtell

 There are various places in the code where we see comments to the effect 
 would be great if we had a scatter gather API for KV, appearing at places 
 where we rewrite KVs on the server, for example in HRegion where we process 
 appends and increments.
 KeyValues are stored in buffers of fixed length. This approach has 
 performance advantages for the common case where KVs are not manipulated on 
 their way from disk to RPC. The disadvantage of this approach is any 
 manipulation of tags requires the creation of a new buffer to hold the 
 result, and a copy of the KV data into the new buffer. Appends and increments 
 are typically a small percentage of overall workload so this has been fine up 
 to now.
  
 KeyValues can now carry metadata known as tags. Tags are stored contiguously 
 with the rest of the KeyValue. Applications wishing to use tags (like per 
 cell security) change the equation by wanting to rewrite KVs significantly 
 more often. 
 We should consider backing KeyValue with an alternative structure that can 
 better support rewriting portions of its data, appends to existing buffers, 
 scatter-gather copies, possibly even copy-on-write.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9794) KeyValues / cells backed by buffer fragments

[
https://issues.apache.org/jira/browse/HBASE-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798191#comment-13798191
]

Andrew Purtell commented on HBASE-9794:
---

Related to HBASE-7320 I'd say

KeyValues / cells backed by buffer fragments

Key: HBASE-9794
URL: https://issues.apache.org/jira/browse/HBASE-9794
Project: HBase
Issue Type: Brainstorming
Reporter: Andrew Purtell

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9794) KeyValues / cells backed by buffer fragments

[
https://issues.apache.org/jira/browse/HBASE-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Purtell updated HBASE-9794:
--

Description:
There are various places in the code where we see comments to the effect would
be great if we had a scatter gather API for KV, appearing at places where we
rewrite KVs on the server, for example in HRegion where we process appends and
increments.

KeyValues are stored in buffers of fixed length. This approach has performance
advantages for the common case where KVs are not manipulated on their way from
disk to RPC. The disadvantage of this approach is any manipulation of the KV
internals then requires the creation of a new buffer to hold the result, and a
copy of the KV data into the new buffer. Appends and increments are typically a
small percentage of overall workload so this has been fine up to now.

We should consider backing KeyValue with an alternative structure that can
better support rewriting portions of its data, appends to existing buffers,
scatter-gather copies, possibly even copy-on-write.

was:
There are various places in the code where we see comments to the effect would
be great if we had a scatter gather API for KV, appearing at places where we
rewrite KVs on the server, for example in HRegion where we process appends and
increments.

KeyValues are stored in buffers of fixed length. This approach has performance
advantages for the common case where KVs are not manipulated on their way from
disk to RPC. The disadvantage of this approach is any manipulation of tags
requires the creation of a new buffer to hold the result, and a copy of the KV
data into the new buffer. Appends and increments are typically a small
percentage of overall workload so this has been fine up to now.

KeyValues / cells backed by buffer fragments

Key: HBASE-9794
URL: https://issues.apache.org/jira/browse/HBASE-9794
Project: HBase
Issue Type: Brainstorming
Reporter: Andrew Purtell

There are various places in the code where we see comments to the effect
would be great if we had a scatter gather API for KV, appearing at places
where we rewrite KVs on the server, for example in HRegion where we process
appends and increments.
KeyValues are stored in buffers of fixed length. This approach has
performance advantages for the common case where KVs are not manipulated on
their way from disk to RPC. The disadvantage of this approach is any
manipulation of the KV internals then requires the creation of a new buffer
to hold the result, and a copy of the KV data into the new buffer. Appends
and increments are typically a small percentage of overall workload so this
has been fine up to now.

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9749) Custom threadpool for Coprocessor obtained HTables


[ 
https://issues.apache.org/jira/browse/HBASE-9749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798192#comment-13798192
 ] 

stack commented on HBASE-9749:
--

bq. External interface and functionality to users is the same, this way just 
minimizes the possible code paths

External Interface is the same?  No external code -- cps or whatever -- will 
use this method that you are changing?  If so, +1 on patch.  Otherwise, 
consider override to minimize need for users to ch ange their code.

 Custom threadpool for Coprocessor obtained HTables
 --

 Key: HBASE-9749
 URL: https://issues.apache.org/jira/browse/HBASE-9749
 Project: HBase
  Issue Type: Improvement
  Components: Coprocessors
Affects Versions: 0.98.0, 0.94.13, 0.96.1
Reporter: Jesse Yates
Assignee: Jesse Yates
 Attachments: hbase-9749-0.94-v0.patch, hbase-9749-0.96-v0.patch, 
 hbase-9749-trunk-v0.patch


 Coprocessors currently can only use the default HTable constructor that takes 
 a single thread-pool.This is overly constrictive for coprocessors that desire 
 tighter control over their resources.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code


 [ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-9754:
--

Attachment: (was: 9754-rp-hregion-v3.txt)

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9749) Custom threadpool for Coprocessor obtained HTables

2013-10-17 Thread Jesse Yates (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798194#comment-13798194
 ] 

Jesse Yates commented on HBASE-9749:


Yup, no change for users. I'll commit shortly.

 Custom threadpool for Coprocessor obtained HTables
 --

 Key: HBASE-9749
 URL: https://issues.apache.org/jira/browse/HBASE-9749
 Project: HBase
  Issue Type: Improvement
  Components: Coprocessors
Affects Versions: 0.98.0, 0.94.13, 0.96.1
Reporter: Jesse Yates
Assignee: Jesse Yates
 Attachments: hbase-9749-0.94-v0.patch, hbase-9749-0.96-v0.patch, 
 hbase-9749-trunk-v0.patch


 Coprocessors currently can only use the default HTable constructor that takes 
 a single thread-pool.This is overly constrictive for coprocessors that desire 
 tighter control over their resources.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9794) KeyValues / cells backed by buffer fragments

[
https://issues.apache.org/jira/browse/HBASE-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798193#comment-13798193
]

Lars Hofhansl commented on HBASE-9794:
--

I am reminded of my writing network protocols in OCaml in a former life :)

KeyValues / cells backed by buffer fragments

Key: HBASE-9794
URL: https://issues.apache.org/jira/browse/HBASE-9794
Project: HBase
Issue Type: Brainstorming
Reporter: Andrew Purtell

There are various places in the code where we see comments to the effect
would be great if we had a scatter gather API for KV, appearing at places
where we rewrite KVs on the server, for example in HRegion where we process
appends and increments.
KeyValues are stored in buffers of fixed length. This approach has
performance advantages for the common case where KVs are not manipulated on
their way from disk to RPC. The disadvantage of this approach is any
manipulation of the KV internals then requires the creation of a new buffer
to hold the result, and a copy of the KV data into the new buffer. Appends
and increments are typically a small percentage of overall workload so this
has been fine up to now.

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Comment Edited] (HBASE-9794) KeyValues / cells backed by buffer fragments


[ 
https://issues.apache.org/jira/browse/HBASE-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798190#comment-13798190
 ] 

Andrew Purtell edited comment on HBASE-9794 at 10/17/13 6:04 PM:
-

Considering alternatives to the single contiguous buffer as we have now, but 
only if needed. The current way should remain the default way.

The kind of KeyValue manipulations desired here are analogous to those 
performed by network stacks in operating systems. 

The BSD mbuf structure is a good example, although it contains a lot of 
particulars to network stacks.

Back when the dinosaurs roamed the earth I worked with a research OS called 
Scout which had (I though) a particularly nice tree based structure for 
composing and decomposing packet buffers. Could be inspiration. From foggy 
memory it had an API like:

{code}
// Return the current length of the message
extern size_t msgLength (Msg m);

// Replace the contents of message 'm' with that of 'other', but without an 
ownership transfer to 'm', any changes will have COW semantics
extern void msgAssign (Msg m, Msg other);

// Replace the contents of message 'm' with the union of 'left' and 'right', 
but without an ownership transfer to 'm', any changes will have COW semantics
extern void msgJoin (Msg m, Msg left, Msg right);

// Remove 'len' bytes from the head of message 'm' into message 'other'
extern void msgBreak (Msg m, Msg other, size_t len);

// Get a contiguous view over 'len' bytes of new storage at the tail of message 
'm', may cause internal tree manipulations and allocations in order to provide 
it
extern void * msgPush (Msg m, size_t len);

// Get a contiguous view over 'len' bytes at the head of message 'm', may cause 
internal tree manipulations and allocations in order to provide it, and remove 
those bytes from the message
extern void * msgPop (Msg m, size_t len);

// Get a contiguous view over 'len' bytes at the head of message 'm', may cause 
internal tree manipulations and allocations in order to provide it, without 
removing those bytes from the message
extern void * msgPeek (Msg m, size_t len);

// Discard 'len' bytes from the head of the message
extern void msgDiscard (Msg m, size_t len);

// Discard 'len' bytes from the tail of the message
extern void msgTruncate (Msg m, size_t len);

// Initialize state for a walk over the tree of buffers for message 'm'
extern void msgWalkInit (MsgWalk w, Msg m);

// Return a view over the contents of the next buffer for message 'm', or the 
first buffer upon first invocation. Does not trigger any tree manipulations or 
allocations
extern void * msgWalkNext (MsgWalk w, size_t *lenp);

// Clean up walk state
extern void msgWalkDone (MsgWalk w);
{code}


was (Author: apurtell):
Considering alternatives to the single contiguous buffer as we have now, but 
only if needed. The current way should remain the default way.

The kind of KeyValue manipulations desired here are analogous to those 
performed by network stacks in operating systems. 

The BSD mbuf structure is a good example, although it contains a lot of 
particulars to network stacks.

Back when the dinosaurs roamed the earth I worked with a research OS called 
Scout which had (I though) a particularly nice tree based structure for 
composing and decomposing packet buffers. Could be inspiration. From foggy 
memory it had an API like:

{code}
// Return the current length of the message
extern size_t msgLength (Msg m);

// Replace the contents of message 'm' with that of 'other', but without an 
ownership transfer to 'm', any changes will have COW semantics
extern void msgAssign (Msg m, Msg other);

// Replace the contents of message 'm' with the union of 'left' and 'write', 
but without an ownership transfer to 'm', any changes will have COW semantics
extern void msgJoin (Msg m, Msg left, Msg right);

// Remove 'len' bytes from the head of message 'm' into message 'other'
extern void msgBreak (Msg m, Msg other, size_t len);

// Get a contiguous view over 'len' bytes of new storage at the tail of message 
'm', may cause internal tree manipulations and allocations in order to provide 
it
extern void * msgPush (Msg m, size_t len);

// Get a contiguous view over 'len' bytes at the head of message 'm', may cause 
internal tree manipulations and allocations in order to provide it, and remove 
those bytes from the message
extern void * msgPop (Msg m, size_t len);

// Get a contiguous view over 'len' bytes at the head of message 'm', may cause 
internal tree manipulations and allocations in order to provide it, without 
removing those bytes from the message
extern void * msgPeek (Msg m, size_t len);

// Discard 'len' bytes from the head of the message
extern void msgDiscard (Msg m, size_t len);

// Discard 'len' bytes from the tail of the message
extern void msgTruncate (Msg m, size_t len);

// Initialize state for a walk over the tree of

[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)


[ 
https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798196#comment-13798196
 ] 

stack commented on HBASE-8741:
--

Try passing -d32 when running that test locally Himanshu.  That might repro the 
issue for you.

Does this last version of the patch address issues raised in last review (there 
was a worry that sequenceids could go in out of order IIRC)

 Scope sequenceid to the region rather than regionserver (WAS: Mutations on 
 Regions in recovery mode might have same sequenceIDs)
 

 Key: HBASE-8741
 URL: https://issues.apache.org/jira/browse/HBASE-8741
 Project: HBase
  Issue Type: Bug
  Components: MTTR
Affects Versions: 0.95.1
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
 Fix For: 0.98.0

 Attachments: HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, 
 HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, 
 HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, 
 HBASE-8741-v5.patch


 Currently, when opening a region, we find the maximum sequence ID from all 
 its HFiles and then set the LogSequenceId of the log (in case the later is at 
 a small value). This works good in recovered.edits case as we are not writing 
 to the region until we have replayed all of its previous edits. 
 With distributed log replay, if we want to enable writes while a region is 
 under recovery, we need to make sure that the logSequenceId  maximum 
 logSequenceId of the old regionserver. Otherwise, we might have a situation 
 where new edits have same (or smaller) sequenceIds. 
 We can store region level information in the WALTrailer, than this scenario 
 could be avoided by:
 a) reading the trailer of the last completed file, i.e., last wal file 
 which has a trailer and,
 b) completely reading the last wal file (this file would not have the 
 trailer, so it needs to be read completely).
 In future, if we switch to multi wal file, we could read the trailer for all 
 completed WAL files, and reading the remaining incomplete files.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code


 [ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-9754:
--

Attachment: 9754-rp-hregion-v3.txt

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt, 9754-rp-hregion-v3.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9791) MR initializes scanner twice


[ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798198#comment-13798198
 ] 

Hadoop QA commented on HBASE-9791:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12608957/trunk-9791.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7578//console

This message is automatically generated.

 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)

[jira] [Commented] (HBASE-4811) Support reverse Scan


[ 
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798200#comment-13798200
 ] 

Lars Hofhansl commented on HBASE-4811:
--

[~apurtell], did you get any chance to do some perf testing on this?
As far as I am concerned this is good now in terms of performance.


 Support reverse Scan
 

 Key: HBASE-4811
 URL: https://issues.apache.org/jira/browse/HBASE-4811
 Project: HBase
  Issue Type: New Feature
  Components: Client
Affects Versions: 0.20.6, 0.94.7
Reporter: John Carrino
Assignee: chunhui shen
 Fix For: 0.98.0

 Attachments: 4811-0.94-v3.txt, 4811-trunk-v10.txt, 
 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94 
 v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch, 
 hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch, 
 hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch, 
 hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch, 
 hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch, 
 hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch, 
 hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch, 
 hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch, 
 hbase-4811-trunkv9.patch


 Reversed scan means scan the rows backward. 
 And StartRow bigger than StopRow in a reversed scan.
 For example, for the following rows:
 aaa/c1:q1/value1
 aaa/c1:q2/value2
 bbb/c1:q1/value1
 bbb/c1:q2/value2
 ccc/c1:q1/value1
 ccc/c1:q2/value2
 ddd/c1:q1/value1
 ddd/c1:q2/value2
 eee/c1:q1/value1
 eee/c1:q2/value2
 you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
 Scan scan = new Scan();
 scan.setStartRow('ddd');
 scan.setStopRow('bbb');
 scan.setReversed(true);
 for(Result result:htable.getScanner(scan)){
  System.out.println(result);
 }
 Aslo you could do the reversed scan with shell like this:
 hbase scan 'table',{REVERSED = true,STARTROW='ddd', STOPROW='bbb'}
 And the output is:
 ddd/c1:q1/value1
 ddd/c1:q2/value2
 ccc/c1:q1/value1
 ccc/c1:q2/value2
 NOTE: when setting reversed as true for a client scan, you must set the start 
 row, else will throw exception. Through {@link 
 Scan#createBiggestByteArray(int)},you could get a big enough byte array as 
 the start row
 All the documentation I find about HBase says that if you want forward and 
 reverse scans you should just build 2 tables and one be ascending and one 
 descending.  Is there a fundamental reason that HBase only supports forward 
 Scan?  It seems like a lot of extra space overhead and coding overhead (to 
 keep them in sync) to support 2 tables.  
 I am assuming this has been discussed before, but I can't find the 
 discussions anywhere about it or why it would be infeasible.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9794) KeyValues / cells backed by buffer fragments

[
https://issues.apache.org/jira/browse/HBASE-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798188#comment-13798188
]

Lars Hofhansl commented on HBASE-9794:
--

This is one of my pet peeves :) and the reason why scanning with block
encoding is so much slower and more GC intensive than without.

KeyValues / cells backed by buffer fragments

Key: HBASE-9794
URL: https://issues.apache.org/jira/browse/HBASE-9794
Project: HBase
Issue Type: Brainstorming
Reporter: Andrew Purtell

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-4811) Support reverse Scan


[ 
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798205#comment-13798205
 ] 

Andrew Purtell commented on HBASE-4811:
---

[~lhofhansl] I'm sadly underwater, if you think it's good +1 from me

 Support reverse Scan
 

 Key: HBASE-4811
 URL: https://issues.apache.org/jira/browse/HBASE-4811
 Project: HBase
  Issue Type: New Feature
  Components: Client
Affects Versions: 0.20.6, 0.94.7
Reporter: John Carrino
Assignee: chunhui shen
 Fix For: 0.98.0

 Attachments: 4811-0.94-v3.txt, 4811-trunk-v10.txt, 
 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94 
 v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch, 
 hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch, 
 hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch, 
 hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch, 
 hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch, 
 hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch, 
 hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch, 
 hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch, 
 hbase-4811-trunkv9.patch


 Reversed scan means scan the rows backward. 
 And StartRow bigger than StopRow in a reversed scan.
 For example, for the following rows:
 aaa/c1:q1/value1
 aaa/c1:q2/value2
 bbb/c1:q1/value1
 bbb/c1:q2/value2
 ccc/c1:q1/value1
 ccc/c1:q2/value2
 ddd/c1:q1/value1
 ddd/c1:q2/value2
 eee/c1:q1/value1
 eee/c1:q2/value2
 you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
 Scan scan = new Scan();
 scan.setStartRow('ddd');
 scan.setStopRow('bbb');
 scan.setReversed(true);
 for(Result result:htable.getScanner(scan)){
  System.out.println(result);
 }
 Aslo you could do the reversed scan with shell like this:
 hbase scan 'table',{REVERSED = true,STARTROW='ddd', STOPROW='bbb'}
 And the output is:
 ddd/c1:q1/value1
 ddd/c1:q2/value2
 ccc/c1:q1/value1
 ccc/c1:q2/value2
 NOTE: when setting reversed as true for a client scan, you must set the start 
 row, else will throw exception. Through {@link 
 Scan#createBiggestByteArray(int)},you could get a big enough byte array as 
 the start row
 All the documentation I find about HBase says that if you want forward and 
 reverse scans you should just build 2 tables and one be ascending and one 
 descending.  Is there a fundamental reason that HBase only supports forward 
 Scan?  It seems like a lot of extra space overhead and coding overhead (to 
 keep them in sync) to support 2 tables.  
 I am assuming this has been discussed before, but I can't find the 
 discussions anywhere about it or why it would be infeasible.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException


[ 
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798209#comment-13798209
 ] 

stack commented on HBASE-9776:
--

[~jeffreyz] you going to commit?

 Test Load And Verify Fails with TableNotEnabledException
 

 Key: HBASE-9776
 URL: https://issues.apache.org/jira/browse/HBASE-9776
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.96.0
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Minor
 Attachments: hbase-9776.patch


 Occasionally IntegrationTestLoadAndVerify failed with the following error. 
 This is caused by RPC retry and the first attempt actually went through 
 successfully and the second retry attempt fails because the table is disabled 
 by the first attempt.
 {code}
 2013-10-10 
 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
  org.apache.hadoop.hbase.TableNotEnabledException: 
 IntegrationTestLoadAndVerify
 2013-10-10 19:55:54,340|beaver.machine|INFO|at 
 org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
 2013-10-10 19:55:54,341|beaver.machine|INFO|at 
 org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
 2013-10-10 19:55:54,342|beaver.machine|INFO|at 
 org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException


[ 
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798216#comment-13798216
 ] 

Jeffrey Zhong commented on HBASE-9776:
--

Yes, I'll commit soon. Thanks.

 Test Load And Verify Fails with TableNotEnabledException
 

 Key: HBASE-9776
 URL: https://issues.apache.org/jira/browse/HBASE-9776
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.96.0
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Minor
 Attachments: hbase-9776.patch


 Occasionally IntegrationTestLoadAndVerify failed with the following error. 
 This is caused by RPC retry and the first attempt actually went through 
 successfully and the second retry attempt fails because the table is disabled 
 by the first attempt.
 {code}
 2013-10-10 
 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
  org.apache.hadoop.hbase.TableNotEnabledException: 
 IntegrationTestLoadAndVerify
 2013-10-10 19:55:54,340|beaver.machine|INFO|at 
 org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
 2013-10-10 19:55:54,341|beaver.machine|INFO|at 
 org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
 2013-10-10 19:55:54,342|beaver.machine|INFO|at 
 org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9749) Custom threadpool for Coprocessor obtained HTables

2013-10-17 Thread Jesse Yates (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-9749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Yates updated HBASE-9749:
---

   Resolution: Fixed
Fix Version/s: 0.94.13
   0.96.0
   0.98.0
   Status: Resolved  (was: Patch Available)

 Custom threadpool for Coprocessor obtained HTables
 --

 Key: HBASE-9749
 URL: https://issues.apache.org/jira/browse/HBASE-9749
 Project: HBase
  Issue Type: Improvement
  Components: Coprocessors
Affects Versions: 0.98.0, 0.94.13, 0.96.1
Reporter: Jesse Yates
Assignee: Jesse Yates
 Fix For: 0.98.0, 0.96.0, 0.94.13

 Attachments: hbase-9749-0.94-v0.patch, hbase-9749-0.96-v0.patch, 
 hbase-9749-trunk-v0.patch


 Coprocessors currently can only use the default HTable constructor that takes 
 a single thread-pool.This is overly constrictive for coprocessors that desire 
 tighter control over their resources.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9794) KeyValues / cells backed by buffer fragments

[
https://issues.apache.org/jira/browse/HBASE-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Purtell updated HBASE-9794:
--

Affects Version/s: 0.98.0

KeyValues / cells backed by buffer fragments

Key: HBASE-9794
URL: https://issues.apache.org/jira/browse/HBASE-9794
Project: HBase
Issue Type: Brainstorming
Affects Versions: 0.98.0
Reporter: Andrew Purtell

There are various places in the code where we see comments to the effect
would be great if we had a scatter gather API for KV, appearing at places
where we rewrite KVs on the server, for example in HRegion where we process
appends and increments.
KeyValues are stored in buffers of fixed length. This approach has
performance advantages for the common case where KVs are not manipulated on
their way from disk to RPC. The disadvantage of this approach is any
manipulation of the KV internals then requires the creation of a new buffer
to hold the result, and a copy of the KV data into the new buffer. Appends
and increments are typically a small percentage of overall workload so this
has been fine up to now.

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9793) Offline a region before it's closed could cause double assignment


[ 
https://issues.apache.org/jira/browse/HBASE-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798232#comment-13798232
 ] 

Jeffrey Zhong commented on HBASE-9793:
--

I reviewed the patch. One minor thing is that we should remove the following 
code. The rest looks good for me. It'll be better if you have a successful IT 
run before to check in. Thanks.
{code}
+if (server.isStopped() || server.isAborted()) {
+  LOG.info(Skip assigning  + region.getRegionNameAsString()
++ , the server is stopped/aborted);
+}
{code}

Because we have in the assign as following. You can add the check  
server.isAborted() though in the loop condition.

{code}
  for (int i = 1; i = maximumAttempts  !server.isStopped(); i++) {
{code}

 Offline a region before it's closed could cause double assignment
 -

 Key: HBASE-9793
 URL: https://issues.apache.org/jira/browse/HBASE-9793
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: trunk-9793.patch


 The fix for HBASE-9773 could cause double assignment, as [~jeffreyz] pointed 
 out. Let's fix it in a separate jira instead of an addendum since there are 
 different opinions on how to fix it.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9747) PrefixFilter with OR condition gives wrong results

2013-10-17 Thread Aditya Kishore (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798239#comment-13798239
 ] 

Aditya Kishore commented on HBASE-9747:
---

{code}
  public boolean filterRowKey(byte[] buffer, int offset, int length) {
...
=  filterRow = (cmp != 0);
return filterRow;
  }
{code}
Because the {{filterRowKey()}} filters a row unless the control reach here ↑.

So it is natural for {{filterRow}} to be {{true}} (initialized and reset for 
every row) unless set to {{false}} by \[{{filterRow = (cmp != 0)}}\] while 
inspecting a row.

 PrefixFilter with OR condition gives wrong results
 --

 Key: HBASE-9747
 URL: https://issues.apache.org/jira/browse/HBASE-9747
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 0.95.2, 0.94.9
Reporter: Deepa Remesh
Assignee: Aditya Kishore
 Attachments: HBASE-9747_0.94.patch, HBASE-9747.patch


 PrefixFilter when used with a SingleColumnValueFilter with an OR condition 
 gives wrong results. In below example, each filter when evaluated separately 
 gives 1 row each. The OR condition with the two filters gives 3 rows instead 
 of 2. Repro below:
 create 't1', 'f1'
 put 't1','a1','f1:q2','111'
 put 't1','b1','f1:q1','112'
 put 't1','c1','f1:q1','113'
 hbase(main):020:0 scan 't1', {FILTER = PrefixFilter ('b') OR 
 SingleColumnValueFilter('f1', 'q1', =, 'binary:113')}
 ROWCOLUMN+CELL
  a1column=f1:q2, 
 timestamp=1381468905492, value=111
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
  c1column=f1:q1, 
 timestamp=1381468905549, value=113
 3 row(s) in 0.1020 seconds
 hbase(main):021:0 scan 't1', {FILTER = PrefixFilter ('b')}
 ROWCOLUMN+CELL
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
 1 row(s) in 0.0150 seconds
 hbase(main):002:0 scan 't1', {FILTER = SingleColumnValueFilter('f1', 'q1', 
 =, 'binary:113')}
 ROWCOLUMN+CELL
  c1column=f1:q1, 
 timestamp=1381469178679, value=113
 1 row(s) in 0.0140 seconds



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9793) Offline a region before it's closed could cause double assignment


[ 
https://issues.apache.org/jira/browse/HBASE-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798244#comment-13798244
 ] 

Jimmy Xiang commented on HBASE-9793:


Good catch. Can we change the loop condition instead so that we can have 
something in the log to find out what's going on?

 Offline a region before it's closed could cause double assignment
 -

 Key: HBASE-9793
 URL: https://issues.apache.org/jira/browse/HBASE-9793
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: trunk-9793.patch


 The fix for HBASE-9773 could cause double assignment, as [~jeffreyz] pointed 
 out. Let's fix it in a separate jira instead of an addendum since there are 
 different opinions on how to fix it.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-8912) [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE

2013-10-17 Thread Vladimir Rodionov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798249#comment-13798249
 ] 

Vladimir Rodionov commented on HBASE-8912:
--

I posted how to reproduce this issue. 
https://issues.apache.org/jira/browse/HBASE-8912?focusedCommentId=13721821page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13721821

 [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to 
 OFFLINE
 --

 Key: HBASE-8912
 URL: https://issues.apache.org/jira/browse/HBASE-8912
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
 Fix For: 0.94.13

 Attachments: HBase-0.94 #1036 test - testRetrying [Jenkins].html


 AM throws this exception which subsequently causes the master to abort: 
 {code}
 java.lang.IllegalStateException: Unexpected state : 
 testRetrying,jjj,1372891751115.9b828792311001062a5ff4b1038fe33b. 
 state=PENDING_OPEN, ts=1372891751912, 
 server=hemera.apache.org,39064,1372891746132 .. Cannot transit it to OFFLINE.
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
   at 
 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
   at 
 org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
   at java.lang.Thread.run(Thread.java:662)
 {code}
 This exception trace is from the failing test TestMetaReaderEditor which is 
 failing pretty frequently, but looking at the test code, I think this is not 
 a test-only issue, but affects the main code path. 
 https://builds.apache.org/job/HBase-0.94/1036/testReport/junit/org.apache.hadoop.hbase.catalog/TestMetaReaderEditor/testRetrying/



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-8143) HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM


[ 
https://issues.apache.org/jira/browse/HBASE-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798257#comment-13798257
 ] 

stack commented on HBASE-8143:
--

I ran into this issue recently and followed Lars advice to fix it.

dfs.client.read.shortcircuit.buffer.size set to 128k all around 
namedfs.client.read.shortcircuit.buffer.size/name value131072/value 

We should add this to our default configs rather than let folks run into OOMEs.

 HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM 
 --

 Key: HBASE-8143
 URL: https://issues.apache.org/jira/browse/HBASE-8143
 Project: HBase
  Issue Type: Bug
  Components: hadoop2
Affects Versions: 0.98.0, 0.94.7, 0.95.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.98.0, 0.96.0, 0.94.13

 Attachments: OpenFileTest.java


 We've run into an issue with HBase 0.94 on Hadoop2, with SSR turned on that 
 the memory usage of the HBase process grows to 7g, on an -Xmx3g, after some 
 time, this causes OOM for the RSs. 
 Upon further investigation, I've found out that we end up with 200 regions, 
 each having 3-4 store files open. Under hadoop2 SSR, BlockReaderLocal 
 allocates DirectBuffers, which is unlike HDFS 1 where there is no direct 
 buffer allocation. 
 It seems that there is no guards against the memory used by local buffers in 
 hdfs 2, and having a large number of open files causes multiple GB of memory 
 to be consumed from the RS process. 
 This issue is to further investigate what is going on. Whether we can limit 
 the memory usage in HDFS, or HBase, and/or document the setup. 
 Possible mitigation scenarios are: 
  - Turn off SSR for Hadoop 2
  - Ensure that there is enough unallocated memory for the RS based on 
 expected # of store files
  - Ensure that there is lower number of regions per region server (hence 
 number of open files)
 Stack trace:
 {code}
 org.apache.hadoop.hbase.DroppedSnapshotException: region: 
 IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946.
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1560)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1439)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1380)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:449)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:215)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:63)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
 at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:632)
 at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:97)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
 at 
 org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(DirectBufferPool.java:70)
 at 
 org.apache.hadoop.hdfs.BlockReaderLocal.init(BlockReaderLocal.java:315)
 at 
 org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:208)
 at 
 org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:790)
 at 
 org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:888)
 at 
 org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:455)
 at 
 org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:645)
 at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689)
 at java.io.DataInputStream.readFully(DataInputStream.java:178)
 at 
 org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:312)
 at 
 org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:543)
 at 
 org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:589)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1261)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:512)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.Store.validateStoreFile(Store.java:1568)
 at 
 org.apache.hadoop.hbase.regionserver.Store.commitFile(Store.java:845)
 at 
 org.apache.hadoop.hbase.regionserver.Store.access$500(Store.java:109)
 at

[jira] [Commented] (HBASE-9792) Region states should update last assignments when a region is opened.


[ 
https://issues.apache.org/jira/browse/HBASE-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798245#comment-13798245
 ] 

Hadoop QA commented on HBASE-9792:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12608965/trunk-9792.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7580//console

This message is automatically generated.

 Region states should update last assignments when a region is opened.
 -

 Key: HBASE-9792
 URL: https://issues.apache.org/jira/browse/HBASE-9792
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: trunk-9792.patch


 Currently, we update a region's last assignment region server when the region 
 is online.  We should do this sooner, when the region is moved to OPEN state. 
  CM could kill this region server before we delete the znode and online the 
 region.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HBASE-9795) Backport the type support

2013-10-17 Thread Diego Masini (JIRA)

Diego Masini created HBASE-9795:
---

 Summary: Backport the type support
 Key: HBASE-9795
 URL: https://issues.apache.org/jira/browse/HBASE-9795
 Project: HBase
  Issue Type: New Feature
  Components: Client
Reporter: Diego Masini


Backport type support from v0.95.2 to 0.94. See HBASE-8089 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9793) Offline a region before it's closed could cause double assignment


[ 
https://issues.apache.org/jira/browse/HBASE-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798261#comment-13798261
 ] 

Jeffrey Zhong commented on HBASE-9793:
--

{quote}
Can we change the loop condition instead so that we can have something in the 
log to find out what's going on?
{quote}
I think that's ok because we already log the info when a server is about to 
abort / stop. Depends on you though.

 Offline a region before it's closed could cause double assignment
 -

 Key: HBASE-9793
 URL: https://issues.apache.org/jira/browse/HBASE-9793
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: trunk-9793.patch


 The fix for HBASE-9773 could cause double assignment, as [~jeffreyz] pointed 
 out. Let's fix it in a separate jira instead of an addendum since there are 
 different opinions on how to fix it.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-8143) HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM


 [ 
https://issues.apache.org/jira/browse/HBASE-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-8143:
-

 Priority: Critical  (was: Major)
Fix Version/s: (was: 0.96.0)
   0.96.1

 HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM 
 --

 Key: HBASE-8143
 URL: https://issues.apache.org/jira/browse/HBASE-8143
 Project: HBase
  Issue Type: Bug
  Components: hadoop2
Affects Versions: 0.98.0, 0.94.7, 0.95.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
 Fix For: 0.98.0, 0.94.13, 0.96.1

 Attachments: OpenFileTest.java


 We've run into an issue with HBase 0.94 on Hadoop2, with SSR turned on that 
 the memory usage of the HBase process grows to 7g, on an -Xmx3g, after some 
 time, this causes OOM for the RSs. 
 Upon further investigation, I've found out that we end up with 200 regions, 
 each having 3-4 store files open. Under hadoop2 SSR, BlockReaderLocal 
 allocates DirectBuffers, which is unlike HDFS 1 where there is no direct 
 buffer allocation. 
 It seems that there is no guards against the memory used by local buffers in 
 hdfs 2, and having a large number of open files causes multiple GB of memory 
 to be consumed from the RS process. 
 This issue is to further investigate what is going on. Whether we can limit 
 the memory usage in HDFS, or HBase, and/or document the setup. 
 Possible mitigation scenarios are: 
  - Turn off SSR for Hadoop 2
  - Ensure that there is enough unallocated memory for the RS based on 
 expected # of store files
  - Ensure that there is lower number of regions per region server (hence 
 number of open files)
 Stack trace:
 {code}
 org.apache.hadoop.hbase.DroppedSnapshotException: region: 
 IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946.
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1560)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1439)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1380)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:449)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:215)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:63)
 at 
 org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
 at java.lang.Thread.run(Thread.java:662)
 Caused by: java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:632)
 at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:97)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
 at 
 org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(DirectBufferPool.java:70)
 at 
 org.apache.hadoop.hdfs.BlockReaderLocal.init(BlockReaderLocal.java:315)
 at 
 org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:208)
 at 
 org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:790)
 at 
 org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:888)
 at 
 org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:455)
 at 
 org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:645)
 at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689)
 at java.io.DataInputStream.readFully(DataInputStream.java:178)
 at 
 org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:312)
 at 
 org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:543)
 at 
 org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:589)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1261)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:512)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:603)
 at 
 org.apache.hadoop.hbase.regionserver.Store.validateStoreFile(Store.java:1568)
 at 
 org.apache.hadoop.hbase.regionserver.Store.commitFile(Store.java:845)
 at 
 org.apache.hadoop.hbase.regionserver.Store.access$500(Store.java:109)
 at 
 org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.commit(Store.java:2209)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1541)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9753) Excessive readpoint checks in MemstoreScanner


[ 
https://issues.apache.org/jira/browse/HBASE-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798265#comment-13798265
 ] 

stack commented on HBASE-9753:
--

Should be final in below

+
+private long readPoint;

And its readPoint here but in other patch it is readPt.

 Excessive readpoint checks in MemstoreScanner
 -

 Key: HBASE-9753
 URL: https://issues.apache.org/jira/browse/HBASE-9753
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0, 0.94.13, 0.96.1

 Attachments: 9753-0.94.txt, 9753-v1.txt


 Brought up by [~vrodionov] on the mailing list. See also HBASE-9751.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9791) MR initializes scanner twice


 [ 
https://issues.apache.org/jira/browse/HBASE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-9791:
---

   Resolution: Fixed
Fix Version/s: 0.96.1
   0.98.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks.  Integrated into trunk and 0.96.

 MR initializes scanner twice
 

 Key: HBASE-9791
 URL: https://issues.apache.org/jira/browse/HBASE-9791
 Project: HBase
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.98.0, 0.96.1

 Attachments: trunk-9791.patch


 The first is in TableInputFormatBase.createRecordReader(). The second time is 
 in initializing it.  We should not call initialize in creating the record 
 reader.
 {noformat}
 2013-10-16 16:58:27,163 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=466730774138784884 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region 
 region=IntegrationTestBigLinkedList,,1381966998140.518fba7c69f069bef99658ca172f9009.,
  hostname=e1521.halxg.cloudera.com,36020,1381967631098, seqNum=968466 
 ip:e1521.halxg.cloudera.com:36020
 2013-10-16 16:58:27,164 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable:org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:312)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:157)
   at 
 org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:116)
   at 
 org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:94)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:176)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:171)
   at 
 org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:110)
   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:719)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.restart(TableRecordReaderImpl.java:86)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.initialize(TableRecordReaderImpl.java:148)
   at 
 org.apache.hadoop.hbase.mapreduce.TableRecordReader.initialize(TableRecordReader.java:125)
   at 
 org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:135)
   at 
 org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:491)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:734)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
 2013-10-16 16:58:27,167 INFO [main] 
 org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: Current 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
 2013-10-16 16:58:27,175 INFO [main] org.apache.hadoop.mapred.MapTask: Map 
 output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
 2013-10-16 16:58:27,891 INFO [main] org.apache.hadoop.mapred.MapTask: 
 (EQUATOR) 0 kvi 268435452(1073741808)
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 mapreduce.task.io.sort.mb: 1024
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: soft 
 limit at 858993472
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: 
 bufstart = 0; bufvoid = 1073741824
 2013-10-16 16:58:27,892 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart 
 = 268435452; length = 67108864
 2013-10-16 16:58:27,903 INFO [main] 
 org.apache.hadoop.hbase.client.ScannerCallable: Open 
 scanner=7462140737850801183 for 
 scan={timeRange:[0,9223372036854775807],batch:-1,startRow:,stopRow:\\x08\\x02\\xC2b,loadColumnFamiliesOnDemand:null,totalColumns:1,cacheBlocks:false,families:{meta:[prev]},maxResultSize:-1,maxVersions:1,caching:100}
  on region

[jira] [Commented] (HBASE-9768) Two issues in AsyncProcess


[ 
https://issues.apache.org/jira/browse/HBASE-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798291#comment-13798291
 ] 

stack commented on HBASE-9768:
--

+1

 Two issues in AsyncProcess
 --

 Key: HBASE-9768
 URL: https://issues.apache.org/jira/browse/HBASE-9768
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Jeffrey Zhong
Assignee: Nicolas Liochon
 Fix For: 0.98.0, 0.96.1

 Attachments: 9768.v1.patch


 There may exist two issues in Asyncprocess code as following:
 1)  In Htable#backgroundFlushCommits, we have following code:
 {code}
   if (ap.hasError()) {
 if (!clearBufferOnFail) {
   // if clearBufferOnFailed is not set, we're supposed to keep the 
 failed operation in the
   //  write buffer. This is a questionable feature kept here for 
 backward compatibility
   writeAsyncBuffer.addAll(ap.getFailedOperations());
 }
 RetriesExhaustedWithDetailsException e = ap.getErrors();
 ap.clearErrors();
 throw e;
   }
 {code}
 In a rare situation like the following: 
 When there are some updates ongoing, a client call Put(internally 
 backgroundFlushCommits get triggered). Then comes the issue:
 The first ap.hasError() returns false and the second ap.hasError() returns 
 true. So we could throw exception to caller while writeAsyncBuffer isn't 
 empty.(some updates are still on going).
 If a client retry with different values for the same keys, we could end up 
 with nondeterministic state.
 2) The following code only update cache for the first row. We should update 
 cache for all the regions inside resultForRS because actions are sent to 
 multiple regions per RS
 {code}
   if (failureCount++ == 0) { // We're doing this once per location.
 hConnection.updateCachedLocations(this.tableName, row.getRow(), 
 result, location);
 if (errorsByServer != null) {
   errorsByServer.reportServerError(location);
   canRetry = errorsByServer.canRetryMore();
 }
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-7525) A canary monitoring program specifically for regionserver


 [ 
https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-7525:
-

Fix Version/s: 0.96.1

 A canary monitoring program specifically for regionserver
 -

 Key: HBASE-7525
 URL: https://issues.apache.org/jira/browse/HBASE-7525
 Project: HBase
  Issue Type: New Feature
  Components: monitoring
Affects Versions: 0.94.0
Reporter: takeshi.miao
Priority: Critical
 Fix For: 0.98.0, 0.96.1

 Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch, 
 HBASE-7525-0.95-v3.patch, HBASE-7525-0.95-v4.patch, HBASE-7525-0.95-v6.patch, 
 HBASE-7525-0.95-v7.patch, HBASE-7525-trunk-v2.patch, 
 HBASE-7525-trunk-v3.patch, HBASE-7525-trunk-v4.patch, HBASE-7525-v0.patch, 
 RegionServerCanary.java


 *Motivation*
 This ticket is to provide a canary monitoring tool specifically for 
 HRegionserver, details as follows
 1. This tool is required by operation team due to they thought that the 
 canary for each region of a HBase is too many for them, so I implemented this 
 coarse-granular one based on the original o.a.h.h.tool.Canary for them
 2. And this tool is implemented by multi-threading, which means the each Get 
 request sent by a thread. the reason I use this way is due to we suffered the 
 region server hung issue by now the root cause is still not clear. so this 
 tool can help operation team to detect hung region server if any.
 *example*
 1. the tool docs
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help
 Usage: [opts] [regionServerName 1 [regionServrName 2...]]
  regionServerName - FQDN serverName, can use linux command:hostname -f to 
 check your serverName
  where [-opts] are:
-help Show this help and exit.
-eUse regionServerName as regular expression
   which means the regionServerName is regular expression pattern
-f B stop whole program if first error occurs, default is true
-t N timeout for a check, default is 60 (milisecs)
-daemonContinuous check at defined intervals.
-interval N  Interval between checks (sec)
 2. Will send a request to each regionserver in a HBase cluster
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary
 3. Will send a request to a regionserver by given name
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname
 4. Will send a request to regionserver(s) by given regular-expression
 /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e 
 rs1.domainname.pattern
 // another example
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e 
 tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org
 5. Will send a request to a regionserver and also set a timeout limit for 
 this test
 // query regionserver:rs1.domainname with timeout limit 10sec
 // -f false, means that will not exit this program even test failed
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1 
 rs1.domainname
 // echo 1 if timeout
 echo $?
 6. Will run as daemon mode, which means it will send request to each 
 regionserver periodically
 ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException


 [ 
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Zhong updated HBASE-9776:
-

   Resolution: Fixed
Fix Version/s: 0.96.1
   0.98.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks for the review and comments! I've integrated the patch into 0.96 and 
trunk.

 Test Load And Verify Fails with TableNotEnabledException
 

 Key: HBASE-9776
 URL: https://issues.apache.org/jira/browse/HBASE-9776
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.96.0
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Minor
 Fix For: 0.98.0, 0.96.1

 Attachments: hbase-9776.patch


 Occasionally IntegrationTestLoadAndVerify failed with the following error. 
 This is caused by RPC retry and the first attempt actually went through 
 successfully and the second retry attempt fails because the table is disabled 
 by the first attempt.
 {code}
 2013-10-10 
 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
  org.apache.hadoop.hbase.TableNotEnabledException: 
 IntegrationTestLoadAndVerify
 2013-10-10 19:55:54,340|beaver.machine|INFO|at 
 org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
 2013-10-10 19:55:54,341|beaver.machine|INFO|at 
 org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
 2013-10-10 19:55:54,342|beaver.machine|INFO|at 
 org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)

2013-10-17 Thread Himanshu Vashishtha (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798296#comment-13798296
]

Himanshu Vashishtha commented on HBASE-8741:

Right, the pre-commit build which failed above was using 32 bit vm. The
original issue was a missing boolean member in heap size report, and I filed
HBASE-9785 for that. Its description explains what was happening in the
precommit build on 32 bit with this patch. Here is a patch that includes that
fix. Lets it get tested on the pre-commit build. Thanks.

Yes, this patch ensures that sequenceIds for a region are always appended to
the WAL in an increasing order.
Also, TestHLog#testConcurrentWrites() runs a test where three threads are
writing to one region concurrently. HLogPerformanceEvaluation verify method
ensures that the records inserted are in order.

Thanks a lot for taking a look.

Scope sequenceid to the region rather than regionserver (WAS: Mutations on
Regions in recovery mode might have same sequenceIDs)

Key: HBASE-8741
URL: https://issues.apache.org/jira/browse/HBASE-8741
Project: HBase
Issue Type: Bug
Components: MTTR
Affects Versions: 0.95.1
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Fix For: 0.98.0

Attachments: HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch,
HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch,
HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch,
HBASE-8741-v5.patch

Currently, when opening a region, we find the maximum sequence ID from all
its HFiles and then set the LogSequenceId of the log (in case the later is at
a small value). This works good in recovered.edits case as we are not writing
to the region until we have replayed all of its previous edits.
With distributed log replay, if we want to enable writes while a region is
under recovery, we need to make sure that the logSequenceId maximum
logSequenceId of the old regionserver. Otherwise, we might have a situation
where new edits have same (or smaller) sequenceIds.
We can store region level information in the WALTrailer, than this scenario
could be avoided by:
a) reading the trailer of the last completed file, i.e., last wal file
which has a trailer and,
b) completely reading the last wal file (this file would not have the
trailer, so it needs to be read completely).
In future, if we switch to multi wal file, we could read the trailer for all
completed WAL files, and reading the remaining incomplete files.

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-8496) Implement tags and the internals of how a tag should look like


 [ 
https://issues.apache.org/jira/browse/HBASE-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-8496:
-

Hadoop Flags: Incompatible change,Reviewed

 Implement tags and the internals of how a tag should look like
 --

 Key: HBASE-8496
 URL: https://issues.apache.org/jira/browse/HBASE-8496
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.98.0, 0.95.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
 Fix For: 0.98.0

 Attachments: Comparison.pdf, HBASE-8496_2.patch, HBASE-8496_3.patch, 
 HBASE-8496_3.patch, HBASE-8496_3.patch, HBASE-8496_4.patch, 
 HBASE-8496_5.patch, HBASE-8496_6.patch, HBASE-8496.patch, 
 Performance_report.xlsx, Tag design.pdf, Tag design_updated.pdf, 
 Tag_In_KV_Buffer_For_reference.patch


 The intent of this JIRA comes from HBASE-7897.
 This would help us to decide on the structure and format of how the tags 
 should look like. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-8496) Implement tags and the internals of how a tag should look like

[
https://issues.apache.org/jira/browse/HBASE-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798309#comment-13798309
]

stack commented on HBASE-8496:
--

Shouldn't this issue have a release note? I marked it as reviewed and an
incompatible change (is that right?). What should we add to the refguide on
tags? Or, we are not ready to add anything there just yet?

On the updated design doc:

There is nothing in the way of altering how tags are currently added, right?
It is just done this way for expediency given so much of the core is still up
on KV.

The below should be 0 or more... right?

Every KV can have 1 or more tags.

hmm... nevermind the rest of the comments. It looks like this design doc.
is a long way from what was implemented. np. We just need a bit of a write up
on what went in before 0.98. Can ignore the below.

On slide #3, every tag has a type byte preceeding it? On slide #3 you don't
say what a tag is? Just a run of bytes?

Oh, so looks like the implementation has deviated from the design, right? OK.
Is it written up in short form anywhere? What was implemented?

The tag structure on slide #5 is different from what is on #3. On #5 it talks
of a tagarray (am I being too literal)?

Are there big changes between hfilev2 and hfilev3? They seem small going by
this design doc.

Implement tags and the internals of how a tag should look like
--

Key: HBASE-8496
URL: https://issues.apache.org/jira/browse/HBASE-8496
Project: HBase
Issue Type: New Feature
Affects Versions: 0.98.0, 0.95.2
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.98.0

Attachments: Comparison.pdf, HBASE-8496_2.patch, HBASE-8496_3.patch,
HBASE-8496_3.patch, HBASE-8496_3.patch, HBASE-8496_4.patch,
HBASE-8496_5.patch, HBASE-8496_6.patch, HBASE-8496.patch,
Performance_report.xlsx, Tag design.pdf, Tag design_updated.pdf,
Tag_In_KV_Buffer_For_reference.patch

The intent of this JIRA comes from HBASE-7897.
This would help us to decide on the structure and format of how the tags
should look like.

--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code


[ 
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798324#comment-13798324
 ] 

Hadoop QA commented on HBASE-9754:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12608987/9754-rp-hregion-v3.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 27 new 
or modified tests.

{color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 
1.0 profile.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:red}-1 site{color}.  The patch appears to cause mvn site goal to 
fail.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/7581//console

This message is automatically generated.

 Consider eliminating threadlocals from MVCC code
 

 Key: HBASE-9754
 URL: https://issues.apache.org/jira/browse/HBASE-9754
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
 Fix For: 0.98.0

 Attachments: 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 
 9754-rp-hregion-v2.txt, 9754-rp-hregion-v3.txt


 Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
 Currently we use ThreadLocals to communicate the current readpoint between a 
 RegionScanner and the Store\{File}Scanner's down the stack.
 Since ThreadLocals are not cheap we should consider whether it is possible to 
 pass the readpoint through the call stack instead.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HBASE-9747) PrefixFilter with OR condition gives wrong results


 [ 
https://issues.apache.org/jira/browse/HBASE-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-9747:
-

   Resolution: Fixed
Fix Version/s: 0.96.1
   0.94.13
   0.98.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to all branches. Thanks Aditya.

 PrefixFilter with OR condition gives wrong results
 --

 Key: HBASE-9747
 URL: https://issues.apache.org/jira/browse/HBASE-9747
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 0.95.2, 0.94.9
Reporter: Deepa Remesh
Assignee: Aditya Kishore
 Fix For: 0.98.0, 0.94.13, 0.96.1

 Attachments: HBASE-9747_0.94.patch, HBASE-9747.patch


 PrefixFilter when used with a SingleColumnValueFilter with an OR condition 
 gives wrong results. In below example, each filter when evaluated separately 
 gives 1 row each. The OR condition with the two filters gives 3 rows instead 
 of 2. Repro below:
 create 't1', 'f1'
 put 't1','a1','f1:q2','111'
 put 't1','b1','f1:q1','112'
 put 't1','c1','f1:q1','113'
 hbase(main):020:0 scan 't1', {FILTER = PrefixFilter ('b') OR 
 SingleColumnValueFilter('f1', 'q1', =, 'binary:113')}
 ROWCOLUMN+CELL
  a1column=f1:q2, 
 timestamp=1381468905492, value=111
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
  c1column=f1:q1, 
 timestamp=1381468905549, value=113
 3 row(s) in 0.1020 seconds
 hbase(main):021:0 scan 't1', {FILTER = PrefixFilter ('b')}
 ROWCOLUMN+CELL
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
 1 row(s) in 0.0150 seconds
 hbase(main):002:0 scan 't1', {FILTER = SingleColumnValueFilter('f1', 'q1', 
 =, 'binary:113')}
 ROWCOLUMN+CELL
  c1column=f1:q1, 
 timestamp=1381469178679, value=113
 1 row(s) in 0.0140 seconds



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9789) Change logging for Coprocessor exec call to trace


[ 
https://issues.apache.org/jira/browse/HBASE-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798381#comment-13798381
 ] 

Hudson commented on HBASE-9789:
---

FAILURE: Integrated in HBase-0.94-security #315 (See 
[https://builds.apache.org/job/HBase-0.94-security/315/])
HBASE-9789 Change logging for Coprocessor exec call to trace (larsh: rev 
1533178)
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


 Change logging for Coprocessor exec call to trace
 -

 Key: HBASE-9789
 URL: https://issues.apache.org/jira/browse/HBASE-9789
 Project: HBase
  Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
 Fix For: 0.94.13

 Attachments: 9789.txt


 Currently both RegionServer and HMaster log (debug) this
 Received dynamic protocol exec call with protocolName class
 on each coprocessor exec call.
 We just filled our RegionServer log with 160gb of these since we're making 
 heavy use of coprocessors.
 I would like to change this to trace. Any objections?



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9747) PrefixFilter with OR condition gives wrong results

2013-10-17 Thread Aditya Kishore (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798378#comment-13798378
 ] 

Aditya Kishore commented on HBASE-9747:
---

Thanks Lars.

 PrefixFilter with OR condition gives wrong results
 --

 Key: HBASE-9747
 URL: https://issues.apache.org/jira/browse/HBASE-9747
 Project: HBase
  Issue Type: Bug
  Components: Filters
Affects Versions: 0.95.2, 0.94.9
Reporter: Deepa Remesh
Assignee: Aditya Kishore
 Fix For: 0.98.0, 0.94.13, 0.96.1

 Attachments: HBASE-9747_0.94.patch, HBASE-9747.patch


 PrefixFilter when used with a SingleColumnValueFilter with an OR condition 
 gives wrong results. In below example, each filter when evaluated separately 
 gives 1 row each. The OR condition with the two filters gives 3 rows instead 
 of 2. Repro below:
 create 't1', 'f1'
 put 't1','a1','f1:q2','111'
 put 't1','b1','f1:q1','112'
 put 't1','c1','f1:q1','113'
 hbase(main):020:0 scan 't1', {FILTER = PrefixFilter ('b') OR 
 SingleColumnValueFilter('f1', 'q1', =, 'binary:113')}
 ROWCOLUMN+CELL
  a1column=f1:q2, 
 timestamp=1381468905492, value=111
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
  c1column=f1:q1, 
 timestamp=1381468905549, value=113
 3 row(s) in 0.1020 seconds
 hbase(main):021:0 scan 't1', {FILTER = PrefixFilter ('b')}
 ROWCOLUMN+CELL
  b1column=f1:q1, 
 timestamp=1381468905518, value=112
 1 row(s) in 0.0150 seconds
 hbase(main):002:0 scan 't1', {FILTER = SingleColumnValueFilter('f1', 'q1', 
 =, 'binary:113')}
 ROWCOLUMN+CELL
  c1column=f1:q1, 
 timestamp=1381469178679, value=113
 1 row(s) in 0.0140 seconds



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HBASE-9749) Custom threadpool for Coprocessor obtained HTables