date:20150813


[ 
https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694821#comment-14694821
 ] 

Hudson commented on HBASE-13985:


FAILURE: Integrated in HBase-0.98 #1079 (See 
[https://builds.apache.org/job/HBase-0.98/1079/])
HBASE-13985 Add configuration to skip validating HFile format when bulk loading 
(Victor Xu) (apurtell: rev 82866bd09c3d75d7ff556827920a93d360c65231)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java


 Add configuration to skip validating HFile format when bulk loading
 ---

 Key: HBASE-13985
 URL: https://issues.apache.org/jira/browse/HBASE-13985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.13
Reporter: Victor Xu
Assignee: Victor Xu
Priority: Minor
  Labels: regionserver
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, 
 HBASE-13985.patch


 When bulk loading millions of HFile into one HTable, checking HFile format is 
 the most time-consuming phase. Maybe we could use a parallel mechanism to 
 increase the speed, but when it comes to millions of HFiles, it may still 
 cost dozens of minutes. So I think it's necessary to add an option for 
 advanced user to bulkload without checking HFile format at all. 
 Of course, the default value of this option should be true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13985) Add configuration to skip validating HFile format when bulk loading


[ 
https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694818#comment-14694818
 ] 

Hudson commented on HBASE-13985:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1032 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1032/])
HBASE-13985 Add configuration to skip validating HFile format when bulk loading 
(Victor Xu) (apurtell: rev 82866bd09c3d75d7ff556827920a93d360c65231)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java


 Add configuration to skip validating HFile format when bulk loading
 ---

 Key: HBASE-13985
 URL: https://issues.apache.org/jira/browse/HBASE-13985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.13
Reporter: Victor Xu
Assignee: Victor Xu
Priority: Minor
  Labels: regionserver
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, 
 HBASE-13985.patch


 When bulk loading millions of HFile into one HTable, checking HFile format is 
 the most time-consuming phase. Maybe we could use a parallel mechanism to 
 increase the speed, but when it comes to millions of HFiles, it may still 
 cost dozens of minutes. So I think it's necessary to add an option for 
 advanced user to bulkload without checking HFile format at all. 
 Of course, the default value of this option should be true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14196) Thrift server idle connection timeout issue


[ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694822#comment-14694822
 ] 

Hudson commented on HBASE-14196:


FAILURE: Integrated in HBase-0.98 #1079 (See 
[https://builds.apache.org/job/HBase-0.98/1079/])
HBASE-14196 Thrift server idle connection timeout issue (Vladimir Rodionov) 
(apurtell: rev a705a7484637c94c1d4c3555c3888cf3908611ce)
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java


 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14196) Thrift server idle connection timeout issue


[ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694819#comment-14694819
 ] 

Hudson commented on HBASE-14196:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1032 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1032/])
HBASE-14196 Thrift server idle connection timeout issue (Vladimir Rodionov) 
(apurtell: rev a705a7484637c94c1d4c3555c3888cf3908611ce)
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java


 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14214) list_labels shouldn't raise ArgumentError if no labels are defined


[ 
https://issues.apache.org/jira/browse/HBASE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694820#comment-14694820
 ] 

Hadoop QA commented on HBASE-14214:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12750068/HBASE-14214.patch
  against master branch at commit a399ac9c4f76dccebb3476c418aac68667dee545.
  ATTACHMENT ID: 12750068

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15081//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15081//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15081//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15081//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15081//console

This message is automatically generated.

 list_labels shouldn't raise ArgumentError if no labels are defined 
 ---

 Key: HBASE-14214
 URL: https://issues.apache.org/jira/browse/HBASE-14214
 Project: HBase
  Issue Type: Bug
Reporter: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 1.3.0

 Attachments: HBASE-14214.patch


 list_labels shouldn't raise ArgumentError if no labels are defined. The empty 
 list is a valid return case, and anyway ArgumentError should be used when 
 arguments to the command are erroneous. Here the command is well formed:
 {noformat}
 hbase(main):001:0 list_labels
 ERROR: No auth label defined
 Here is some help for this command:
 List the visibility labels defined in the system.
 Optional regular expression parameter could be used to filter the labels 
 being returned.
 Syntax : list_labels
 For example:
 hbase list_labels 'secret.*'
 hbase list_labels
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14203) remove duplicate code getTableDescriptor in HTable

2015-08-13 Thread Heng Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694838#comment-14694838
 ] 

Heng Chen commented on HBASE-14203:
---

[~enis] any  suggestions?

 remove duplicate code getTableDescriptor in HTable
 --

 Key: HBASE-14203
 URL: https://issues.apache.org/jira/browse/HBASE-14203
 Project: HBase
  Issue Type: Improvement
Reporter: Heng Chen
Priority: Trivial
 Attachments: HBASE-14203.patch, HBASE-14203_v2.patch, 
 HBASE-14203_v3.patch, HBASE-14203_v4.patch


 As TODO in comment said, 
 {{HTable.getTableDescriptor}} is same as {{HAdmin.getTableDescriptor}}. 
 remove the duplicate code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14054) Acknowledged writes may get lost if regionserver clock is set backwards

2015-08-13 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694922#comment-14694922
]

Enis Soztutar commented on HBASE-14054:
---

BTW, test failures are not related. TestImportExport takes ~5 min on my laptop,
something to look at.

Acknowledged writes may get lost if regionserver clock is set backwards
---

Key: HBASE-14054
URL: https://issues.apache.org/jira/browse/HBASE-14054
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.6
Environment: Linux
Reporter: Tobi Vollebregt
Assignee: Enis Soztutar
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0

Attachments: hbase-14054_v1.patch

We experience a small amount of lost acknowledged writes in production on
July 1st (~700 identified so far).
What happened was that we had NTP turned off since June 29th to prevent
issues due to the leap second on June 30th. NTP was turned back on July 1st.
The next day, we noticed we were missing writes to a few of our higher
throughput aggregation tables.
We found that this is caused by HBase taking the current time using
System.currentTimeMillis, which may be set backwards by NTP, and using this
without any checks to populate the timestamp of rows for which the client
didn't supply a timestamp.
Our application uses a read-modify-write pattern using get+checkAndPut to
perform aggregation as follows:
1. read version 1
2. mutate
3. write version 2
4. read version 2
5. mutate
6. write version 3
The application retries the full read-modify-write if the checkAndPut fails.
What must have happened on July 1st, after we started NTP back up, was this
(timestamps added):
1. read version 1 (timestamp 10)
2. mutate
3. write version 2 (HBase-assigned timestamp 11)
4. read version 2 (timestamp 11)
5. mutate
6. write version 3 (HBase-assigned timestamp 10)
Hence, the last write was eclipsed by the first write, and hence, an
acknowledged write was lost.
While this seems to match documented behavior (paraphrasing: if timestamp is
not specified HBase will assign a timestamp using System.currentTimeMillis
the row with the highest timestamp will be returned by get), I think it is
very unintuitive and needs at least a big warning in the documentation, along
the lines of Acknowledged writes may not be visible unless the timestamp is
explicitly specified and equal to or larger than the highest timestamp for
that row.
I would also like to use this ticket to start a discussion on if we can make
the behavior better:
Could HBase assign a timestamp of {{max(max timestamp for the row,
System.currentTimeMillis())}} in the checkAndPut write path, instead of
blindly taking {{System.currentTimeMillis()}}, similar to what has been done
in HBASE-12449 for increment and append?
Thoughts?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14144) Bloomfilter path to work with Byte buffered cells


[ 
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694901#comment-14694901
 ] 

Hadoop QA commented on HBASE-14144:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12750237/HBASE-14144_3.patch
  against master branch at commit a399ac9c4f76dccebb3476c418aac68667dee545.
  ATTACHMENT ID: 12750237

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15082//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15082//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15082//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15082//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15082//console

This message is automatically generated.

 Bloomfilter path to work with Byte buffered cells
 -

 Key: HBASE-14144
 URL: https://issues.apache.org/jira/browse/HBASE-14144
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 2.0.0

 Attachments: HBASE-14144.patch, HBASE-14144_1.patch, 
 HBASE-14144_2.patch, HBASE-14144_3.patch


 This JIRA is to check if there will be a need to make the bloom filters to 
 work with ByteBuffer cells. During POC this path created lot of duplicated 
 code but considering other refactorings done in this path  may lead to less 
 duplication. This JIRA is a placeholder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions


[ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694817#comment-14694817
 ] 

Hudson commented on HBASE-14098:


SUCCESS: Integrated in HBase-1.3-IT #87 (See 
[https://builds.apache.org/job/HBase-1.3-IT/87/])
HBASE-14098 Allow dropping caches behind compactions (eclark: rev 
0dbaf0ea5145b318b18b487d8d9bc2d2eb67e19d)
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java


 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13985) Add configuration to skip validating HFile format when bulk loading


[ 
https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694814#comment-14694814
 ] 

Hudson commented on HBASE-13985:


SUCCESS: Integrated in HBase-1.3-IT #87 (See 
[https://builds.apache.org/job/HBase-1.3-IT/87/])
HBASE-13985 Add configuration to skip validating HFile format when bulk loading 
(Victor Xu) (apurtell: rev ca19f961a25dce5359bfb9b35c0bbbd64ec0fb0b)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java


 Add configuration to skip validating HFile format when bulk loading
 ---

 Key: HBASE-13985
 URL: https://issues.apache.org/jira/browse/HBASE-13985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.13
Reporter: Victor Xu
Assignee: Victor Xu
Priority: Minor
  Labels: regionserver
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, 
 HBASE-13985.patch


 When bulk loading millions of HFile into one HTable, checking HFile format is 
 the most time-consuming phase. Maybe we could use a parallel mechanism to 
 increase the speed, but when it comes to millions of HFiles, it may still 
 cost dozens of minutes. So I think it's necessary to add an option for 
 advanced user to bulkload without checking HFile format at all. 
 Of course, the default value of this option should be true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14122) Client API for determining if server side supports cell level security


[ 
https://issues.apache.org/jira/browse/HBASE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694816#comment-14694816
 ] 

Hudson commented on HBASE-14122:


SUCCESS: Integrated in HBase-1.3-IT #87 (See 
[https://builds.apache.org/job/HBase-1.3-IT/87/])
HBASE-14122 Client API for determining if server side supports cell level 
security (apurtell: rev ba4cc6321b9ed209393dfbc01d2c802062bb7437)
* hbase-shell/src/main/ruby/shell/commands/list_security_capabilities.rb
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
* hbase-shell/src/main/ruby/shell.rb
* hbase-shell/src/main/ruby/hbase/security.rb
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* hbase-shell/src/main/ruby/hbase/visibility_labels.rb
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityClient.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
* hbase-protocol/src/main/protobuf/Master.proto
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/security/SecurityCapability.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlClient.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java
* hbase-shell/src/main/ruby/hbase/admin.rb
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java


 Client API for determining if server side supports cell level security
 --

 Key: HBASE-14122
 URL: https://issues.apache.org/jira/browse/HBASE-14122
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-14122-0.98.patch, HBASE-14122-0.98.patch, 
 HBASE-14122-branch-1.patch, HBASE-14122-branch-1.patch, HBASE-14122.patch, 
 HBASE-14122.patch, HBASE-14122.patch


 Add a client API for determining if the server side supports cell level 
 security. 
 Ask the master, assuming as we do in many other instances that the master and 
 regionservers all have a consistent view of site configuration.
 Return {{true}} if all features required for cell level security are present, 
 {{false}} otherwise, or throw {{UnsupportedOperationException}} if the master 
 does not have support for the RPC call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14196) Thrift server idle connection timeout issue


[ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694815#comment-14694815
 ] 

Hudson commented on HBASE-14196:


SUCCESS: Integrated in HBase-1.3-IT #87 (See 
[https://builds.apache.org/job/HBase-1.3-IT/87/])
HBASE-14196 Thrift server idle connection timeout issue (Vladimir Rodionov) 
(apurtell: rev e8b5e922cb1741ccc4954ef2ba0f97b5d6d39079)
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java


 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14054) Acknowledged writes may get lost if regionserver clock is set backwards

2015-08-13 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Enis Soztutar updated HBASE-14054:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)

I've committed this to 0.98+. Thanks Tobi for reporting, and Andrew for review.

Acknowledged writes may get lost if regionserver clock is set backwards
---

Attachments: hbase-14054_v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14214) list_labels shouldn't raise ArgumentError if no labels are defined


 [ 
https://issues.apache.org/jira/browse/HBASE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-14214:
---
Resolution: Fixed
  Assignee: Anoop Sam John
Status: Resolved  (was: Patch Available)

Pushed to branch-1 and master. Thanks for the review Andy.

 list_labels shouldn't raise ArgumentError if no labels are defined 
 ---

 Key: HBASE-14214
 URL: https://issues.apache.org/jira/browse/HBASE-14214
 Project: HBase
  Issue Type: Bug
Reporter: Andrew Purtell
Assignee: Anoop Sam John
Priority: Minor
 Fix For: 2.0.0, 1.3.0

 Attachments: HBASE-14214.patch


 list_labels shouldn't raise ArgumentError if no labels are defined. The empty 
 list is a valid return case, and anyway ArgumentError should be used when 
 arguments to the command are erroneous. Here the command is well formed:
 {noformat}
 hbase(main):001:0 list_labels
 ERROR: No auth label defined
 Here is some help for this command:
 List the visibility labels defined in the system.
 Optional regular expression parameter could be used to filter the labels 
 being returned.
 Syntax : list_labels
 For example:
 hbase list_labels 'secret.*'
 hbase list_labels
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13376) Improvements to Stochastic load balancer

2015-08-13 Thread Vandana Ayyalasomayajula (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694831#comment-14694831
 ] 

Vandana Ayyalasomayajula commented on HBASE-13376:
--

The last build seems to finally have the tests run:
{quote}
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 506.45 sec - in 
org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 359.237 sec - 
in org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer
{quote}

I cannot seem to find the javadoc warning, that I have introduced :(. 

 Improvements to Stochastic load balancer
 

 Key: HBASE-13376
 URL: https://issues.apache.org/jira/browse/HBASE-13376
 Project: HBase
  Issue Type: Improvement
  Components: Balancer
Affects Versions: 1.0.0, 0.98.12
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
 Attachments: 13376-v2.txt, HBASE-13376.patch, HBASE-13376_0.98.txt, 
 HBASE-13376_0.txt, HBASE-13376_1.txt, HBASE-13376_1_1.txt, 
 HBASE-13376_2.patch, HBASE-13376_2_branch-1.patch, HBASE-13376_3.patch, 
 HBASE-13376_3.patch, HBASE-13376_4.patch, HBASE-13376_98.patch, 
 HBASE-13376_branch-1.patch


 There are two things this jira tries to address:
 1. The locality picker in the stochastic balancer does not pick regions with 
 least locality as candidates for swap/move. So when any user configures 
 locality cost in the configs, the balancer does not always seems to move 
 regions with bad locality. 
 2. When a cluster has equal number of loaded regions, it always picks the 
 first one. It should pick a random region on one of the equally loaded 
 servers. This improves a chance of finding a good candidate, when load picker 
 is invoked several times. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14144) Bloomfilter path to work with Byte buffered cells


[ 
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694858#comment-14694858
 ] 

Anoop Sam John commented on HBASE-14144:


FirstOnRowBBFakeCell 
Need to override getRowArray also as the super class returns empty byte[]. We 
can use CellUtil clone API

FirstOnRowColBBFakeCell 
Same way override getQualifierArray

{code}
Cell kvKey = KeyValueUtil.createFirstOnRow(row, 0, row.length, 
1252  HConstants.EMPTY_BYTE_ARRAY, 0, 0, column, 0,
1253  column.length);
1254return passesGeneralRowColBloomFilter(kvKey);
{code}
Here we are creating a KV by copy and later down in 
checkGeneralBloomFilter(byte[] key, Cell kvKey, BloomFilter bloomFilter)  we 
again create
Cell rowBloomKey = CellUtil.createFirstOnRow(kvKey);
Can we avoid? Some sort of refactoring may be needed. Any chance to use 
CellUtil instead of KeyValueUtil and avoid a copy?

We may have to change Hash function to work with BB also? Later in another jira 
may be

FirstOnRowBBFakeCell  - Better we can name it as FirstOnRowFakeBBCell 


 Bloomfilter path to work with Byte buffered cells
 -

 Key: HBASE-14144
 URL: https://issues.apache.org/jira/browse/HBASE-14144
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 2.0.0

 Attachments: HBASE-14144.patch, HBASE-14144_1.patch, 
 HBASE-14144_2.patch, HBASE-14144_3.patch


 This JIRA is to check if there will be a need to make the bloom filters to 
 work with ByteBuffer cells. During POC this path created lot of duplicated 
 code but considering other refactorings done in this path  may lead to less 
 duplication. This JIRA is a placeholder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14214) list_labels shouldn't raise ArgumentError if no labels are defined


[ 
https://issues.apache.org/jira/browse/HBASE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694919#comment-14694919
 ] 

Anoop Sam John commented on HBASE-14214:


No tests failed. TestShell passed locally. Will commit shortly.

 list_labels shouldn't raise ArgumentError if no labels are defined 
 ---

 Key: HBASE-14214
 URL: https://issues.apache.org/jira/browse/HBASE-14214
 Project: HBase
  Issue Type: Bug
Reporter: Andrew Purtell
Priority: Minor
 Fix For: 2.0.0, 1.3.0

 Attachments: HBASE-14214.patch


 list_labels shouldn't raise ArgumentError if no labels are defined. The empty 
 list is a valid return case, and anyway ArgumentError should be used when 
 arguments to the command are erroneous. Here the command is well formed:
 {noformat}
 hbase(main):001:0 list_labels
 ERROR: No auth label defined
 Here is some help for this command:
 List the visibility labels defined in the system.
 Optional regular expression parameter could be used to filter the labels 
 being returned.
 Syntax : list_labels
 For example:
 hbase list_labels 'secret.*'
 hbase list_labels
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions


[ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694807#comment-14694807
 ] 

Hudson commented on HBASE-14098:


SUCCESS: Integrated in HBase-1.2-IT #88 (See 
[https://builds.apache.org/job/HBase-1.2-IT/88/])
HBASE-14098 Allow dropping caches behind compactions (eclark: rev 
34b706af4d44ad7dff8ac5f35eec304d7dc0ccab)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java


 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13985) Add configuration to skip validating HFile format when bulk loading


[ 
https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694805#comment-14694805
 ] 

Hudson commented on HBASE-13985:


SUCCESS: Integrated in HBase-1.2-IT #88 (See 
[https://builds.apache.org/job/HBase-1.2-IT/88/])
HBASE-13985 Add configuration to skip validating HFile format when bulk loading 
(Victor Xu) (apurtell: rev 222102196f358667462dfbeeb8a63b6826ece397)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java


 Add configuration to skip validating HFile format when bulk loading
 ---

 Key: HBASE-13985
 URL: https://issues.apache.org/jira/browse/HBASE-13985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.13
Reporter: Victor Xu
Assignee: Victor Xu
Priority: Minor
  Labels: regionserver
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, 
 HBASE-13985.patch


 When bulk loading millions of HFile into one HTable, checking HFile format is 
 the most time-consuming phase. Maybe we could use a parallel mechanism to 
 increase the speed, but when it comes to millions of HFiles, it may still 
 cost dozens of minutes. So I think it's necessary to add an option for 
 advanced user to bulkload without checking HFile format at all. 
 Of course, the default value of this option should be true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14196) Thrift server idle connection timeout issue


[ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694806#comment-14694806
 ] 

Hudson commented on HBASE-14196:


SUCCESS: Integrated in HBase-1.2-IT #88 (See 
[https://builds.apache.org/job/HBase-1.2-IT/88/])
HBASE-14196 Thrift server idle connection timeout issue (Vladimir Rodionov) 
(apurtell: rev 93d6fbe9271e85ce144b47cdaeb4bf63162dde37)
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java


 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting

2015-08-13 Thread Heng Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14189:
--
Attachment: HBASE-14189_v1.patch

{quote}
This change has to any way be backward compatible also. Means we should 
continue to support global level setting (for cache data in compressed form)
{quote}

Fine,  i agree with you.   I will continue to support global level setting 
options.

Update the patch

 BlockCache options should consider CF Level BlockCacheEnabled setting
 -

 Key: HBASE-14189
 URL: https://issues.apache.org/jira/browse/HBASE-14189
 Project: HBase
  Issue Type: Improvement
  Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
 Attachments: HBASE-14189.patch, HBASE-14189_v1.patch


 While using BlockCache,  we use {{cacheDataOnRead}}({{cacheDataOnWrite}}) 
 represents for whether we should cache block after read(write) block from(to) 
 hdfs.  We should honour BC setting and CF Level cache setting while using 
 BlockCache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions


[ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694834#comment-14694834
 ] 

Hudson commented on HBASE-14098:


FAILURE: Integrated in HBase-1.2 #106 (See 
[https://builds.apache.org/job/HBase-1.2/106/])
HBASE-14098 Allow dropping caches behind compactions (eclark: rev 
34b706af4d44ad7dff8ac5f35eec304d7dc0ccab)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeCompactor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java


 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13985) Add configuration to skip validating HFile format when bulk loading


[ 
https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694833#comment-14694833
 ] 

Hudson commented on HBASE-13985:


FAILURE: Integrated in HBase-1.2 #106 (See 
[https://builds.apache.org/job/HBase-1.2/106/])
HBASE-13985 Add configuration to skip validating HFile format when bulk loading 
(Victor Xu) (apurtell: rev 222102196f358667462dfbeeb8a63b6826ece397)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java


 Add configuration to skip validating HFile format when bulk loading
 ---

 Key: HBASE-13985
 URL: https://issues.apache.org/jira/browse/HBASE-13985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.13
Reporter: Victor Xu
Assignee: Victor Xu
Priority: Minor
  Labels: regionserver
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, 
 HBASE-13985.patch


 When bulk loading millions of HFile into one HTable, checking HFile format is 
 the most time-consuming phase. Maybe we could use a parallel mechanism to 
 increase the speed, but when it comes to millions of HFiles, it may still 
 cost dozens of minutes. So I think it's necessary to add an option for 
 advanced user to bulkload without checking HFile format at all. 
 Of course, the default value of this option should be true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14054) Acknowledged writes may get lost if regionserver clock is set backwards

[
https://issues.apache.org/jira/browse/HBASE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695008#comment-14695008
]

Hudson commented on HBASE-14054:

FAILURE: Integrated in HBase-1.3-IT #88 (See
[https://builds.apache.org/job/HBase-1.3-IT/88/])
HBASE-14054 Acknowledged writes may get lost if regionserver clock is set
backwards (enis: rev a0086e97ad1966c2c59fde832ed60dbab9ca9e43)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java

Acknowledged writes may get lost if regionserver clock is set backwards
---

Attachments: hbase-14054_v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14054) Acknowledged writes may get lost if regionserver clock is set backwards

[
https://issues.apache.org/jira/browse/HBASE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695095#comment-14695095
]

Hudson commented on HBASE-14054:

FAILURE: Integrated in HBase-0.98 #1080 (See
[https://builds.apache.org/job/HBase-0.98/1080/])
HBASE-14054 Acknowledged writes may get lost if regionserver clock is set
backwards (enis: rev a3914d66c3b9e84f624a1c8718ccd69fb27fe311)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java

Acknowledged writes may get lost if regionserver clock is set backwards
---

Attachments: hbase-14054_v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14144) Bloomfilter path to work with Byte buffered cells


 [ 
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14144:
---
Status: Open  (was: Patch Available)

 Bloomfilter path to work with Byte buffered cells
 -

 Key: HBASE-14144
 URL: https://issues.apache.org/jira/browse/HBASE-14144
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 2.0.0

 Attachments: HBASE-14144.patch, HBASE-14144_1.patch, 
 HBASE-14144_2.patch, HBASE-14144_3.patch, HBASE-14144_4.patch


 This JIRA is to check if there will be a need to make the bloom filters to 
 work with ByteBuffer cells. During POC this path created lot of duplicated 
 code but considering other refactorings done in this path  may lead to less 
 duplication. This JIRA is a placeholder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14144) Bloomfilter path to work with Byte buffered cells


 [ 
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14144:
---
Attachment: HBASE-14144_4.patch

Updated patch that adds CloneXXX methods in the FakeBBCell.


 Bloomfilter path to work with Byte buffered cells
 -

 Key: HBASE-14144
 URL: https://issues.apache.org/jira/browse/HBASE-14144
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 2.0.0

 Attachments: HBASE-14144.patch, HBASE-14144_1.patch, 
 HBASE-14144_2.patch, HBASE-14144_3.patch, HBASE-14144_4.patch


 This JIRA is to check if there will be a need to make the bloom filters to 
 work with ByteBuffer cells. During POC this path created lot of duplicated 
 code but considering other refactorings done in this path  may lead to less 
 duplication. This JIRA is a placeholder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14054) Acknowledged writes may get lost if regionserver clock is set backwards

[
https://issues.apache.org/jira/browse/HBASE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695046#comment-14695046
]

Hudson commented on HBASE-14054:

FAILURE: Integrated in HBase-TRUNK #6724 (See
[https://builds.apache.org/job/HBase-TRUNK/6724/])
HBASE-14054 Acknowledged writes may get lost if regionserver clock is set
backwards (enis: rev d1262331eb0481ecc128ce78590ca4ff992f94e7)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java

Acknowledged writes may get lost if regionserver clock is set backwards
---

Attachments: hbase-14054_v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14054) Acknowledged writes may get lost if regionserver clock is set backwards

[
https://issues.apache.org/jira/browse/HBASE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695024#comment-14695024
]

Hudson commented on HBASE-14054:

SUCCESS: Integrated in HBase-1.0 #1014 (See
[https://builds.apache.org/job/HBase-1.0/1014/])
HBASE-14054 Acknowledged writes may get lost if regionserver clock is set
backwards (enis: rev 50a557193b9a9634891d19174c27e1cd34ff9e6c)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java

Acknowledged writes may get lost if regionserver clock is set backwards
---

Attachments: hbase-14054_v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-6721) RegionServer Group based Assignment

2015-08-13 Thread Francis Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francis Liu updated HBASE-6721:
---
Attachment: HBASE-6721_98_1.patch

draft of backported patch to 98.  need to run integration and unit tests then 
it should be good to go.

 RegionServer Group based Assignment
 ---

 Key: HBASE-6721
 URL: https://issues.apache.org/jira/browse/HBASE-6721
 Project: HBase
  Issue Type: New Feature
Reporter: Francis Liu
Assignee: Francis Liu
 Attachments: 6721-master-webUI.patch, HBASE-6721 
 GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, 
 HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, 
 HBASE-6721_10.patch, HBASE-6721_11.patch, HBASE-6721_8.patch, 
 HBASE-6721_9.patch, HBASE-6721_9.patch, HBASE-6721_94.patch, 
 HBASE-6721_94.patch, HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, 
 HBASE-6721_94_3.patch, HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, 
 HBASE-6721_94_6.patch, HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, 
 HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, 
 HBASE-6721_trunk1.patch, HBASE-6721_trunk2.patch, balanceCluster Sequence 
 Diagram.svg, immediateAssignments Sequence Diagram.svg, randomAssignment 
 Sequence Diagram.svg, retainAssignment Sequence Diagram.svg, 
 roundRobinAssignment Sequence Diagram.svg


 In multi-tenant deployments of HBase, it is likely that a RegionServer will 
 be serving out regions from a number of different tables owned by various 
 client applications. Being able to group a subset of running RegionServers 
 and assign specific tables to it, provides a client application a level of 
 isolation and resource allocation.
 The proposal essentially is to have an AssignmentManager which is aware of 
 RegionServer groups and assigns tables to region servers based on groupings. 
 Load balancing will occur on a per group basis as well. 
 This is essentially a simplification of the approach taken in HBASE-4120. See 
 attached document.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment

[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695083#comment-14695083
]

Hadoop QA commented on HBASE-6721:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12750267/HBASE-6721_98_1.patch
against master branch at commit 9c69bf766fcad024bef5760f242cae2bc609b374.
ATTACHMENT ID: 12750267

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 33 new
or modified tests.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15084//console

This message is automatically generated.

RegionServer Group based Assignment
---

Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Francis Liu
Attachments: 6721-master-webUI.patch, HBASE-6721
GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf,
HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf,
HBASE-6721_10.patch, HBASE-6721_11.patch, HBASE-6721_8.patch,
HBASE-6721_9.patch, HBASE-6721_9.patch, HBASE-6721_94.patch,
HBASE-6721_94.patch, HBASE-6721_94_2.patch, HBASE-6721_94_3.patch,
HBASE-6721_94_3.patch, HBASE-6721_94_4.patch, HBASE-6721_94_5.patch,
HBASE-6721_94_6.patch, HBASE-6721_94_7.patch, HBASE-6721_98_1.patch,
HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk.patch,
HBASE-6721_trunk1.patch, HBASE-6721_trunk2.patch, balanceCluster Sequence
Diagram.svg, immediateAssignments Sequence Diagram.svg, randomAssignment
Sequence Diagram.svg, retainAssignment Sequence Diagram.svg,
roundRobinAssignment Sequence Diagram.svg

In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment

2015-08-13 Thread Francis Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695050#comment-14695050
 ] 

Francis Liu commented on HBASE-6721:


{quote}
1.  Within the interface GroupInfoManager.java, I noticed that the function 
getGroupOfServer returns a GroupInfo object, but the function getGroupOfTable 
returns a String object. Was there a performance consideration or some other 
reason for returning a string? (It seems the API would be more consistent to 
return a GroupInfo object.)
{quote}
Yeah performance. I didn't want to have to gather the information to generate 
the GroupInfo object since most internal calls won't need it. The external api 
call is consistent tho.

{quote}
2.  For the function onlineRetainAssignment why are regions assigned to 
bogus so it ends up in RIT if a server is not available? (We would like to keep 
as few regions in RIT as possible in order to maximize our availability.)
{quote}
This guarantees isolation. Which is one of the key features of Region Server 
Groups. ie you don't want the reason that the region from one group to run out 
of live servers to affect the regions in another group. So worst case you 
affect the availability of the group and not the entire cluster. What's your 
use case for region server groups?

{quote}
3.  On the topic of onlineRetainAssignment, what is the objective for 
separating online and offline servers? I noticed that another balancer such as 
the StochasticLoadBalancer does not make such a distinction.
{quote}
Yes that's inherent in GroupBaseLoadBalancer. Stochastic only looks at what's 
online, while Group needs to look at which member servers are online.


 RegionServer Group based Assignment
 ---

 Key: HBASE-6721
 URL: https://issues.apache.org/jira/browse/HBASE-6721
 Project: HBase
  Issue Type: New Feature
Reporter: Francis Liu
Assignee: Francis Liu
 Attachments: 6721-master-webUI.patch, HBASE-6721 
 GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, 
 HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, 
 HBASE-6721_10.patch, HBASE-6721_11.patch, HBASE-6721_8.patch, 
 HBASE-6721_9.patch, HBASE-6721_9.patch, HBASE-6721_94.patch, 
 HBASE-6721_94.patch, HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, 
 HBASE-6721_94_3.patch, HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, 
 HBASE-6721_94_6.patch, HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, 
 HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, 
 HBASE-6721_trunk1.patch, HBASE-6721_trunk2.patch, balanceCluster Sequence 
 Diagram.svg, immediateAssignments Sequence Diagram.svg, randomAssignment 
 Sequence Diagram.svg, retainAssignment Sequence Diagram.svg, 
 roundRobinAssignment Sequence Diagram.svg


 In multi-tenant deployments of HBase, it is likely that a RegionServer will 
 be serving out regions from a number of different tables owned by various 
 client applications. Being able to group a subset of running RegionServers 
 and assign specific tables to it, provides a client application a level of 
 isolation and resource allocation.
 The proposal essentially is to have an AssignmentManager which is aware of 
 RegionServer groups and assigns tables to region servers based on groupings. 
 Load balancing will occur on a per group basis as well. 
 This is essentially a simplification of the approach taken in HBASE-4120. See 
 attached document.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14144) Bloomfilter path to work with Byte buffered cells


[ 
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694946#comment-14694946
 ] 

ramkrishna.s.vasudevan commented on HBASE-14144:


BTW
bq.-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
This seems unrelated to the patch.
{code}
1 warning
[WARNING] Javadoc Warnings
[WARNING] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java:3147:
 warning - @param argument caps is not a parameter name.
{code}
Should be because of this.  

 Bloomfilter path to work with Byte buffered cells
 -

 Key: HBASE-14144
 URL: https://issues.apache.org/jira/browse/HBASE-14144
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 2.0.0

 Attachments: HBASE-14144.patch, HBASE-14144_1.patch, 
 HBASE-14144_2.patch, HBASE-14144_3.patch, HBASE-14144_4.patch


 This JIRA is to check if there will be a need to make the bloom filters to 
 work with ByteBuffer cells. During POC this path created lot of duplicated 
 code but considering other refactorings done in this path  may lead to less 
 duplication. This JIRA is a placeholder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14144) Bloomfilter path to work with Byte buffered cells

[
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694945#comment-14694945
]

ramkrishna.s.vasudevan commented on HBASE-14144:

bq.Here we are creating a KV by copy and later down in
checkGeneralBloomFilter(byte[] key, Cell kvKey, BloomFilter bloomFilter) we
again create
Cell rowBloomKey = CellUtil.createFirstOnRow(kvKey);
Can we avoid? Some sort of refactoring may be needed. Any chance to use
CellUtil instead of KeyValueUtil and avoid a copy?

The rowBloomKey is needed here for the Deletefamily bloom cases. So the kvKey
is a cell created out of row and col and in this rowBloomKey case we only need
the row part of the Cell. Also the
{code}
exists =
bloomFilter.contains(kvKey, bloom) ||
bloomFilter.contains(rowBloomKey, bloom);
{code}
Will expect a Cell and that is why converted the rowBloomKey to a new Cell.
bq.FirstOnRowBBFakeCell - Better we can name it as FirstOnRowFakeBBCell
Yes changed.
bq.We may have to change Hash function to work with BB also? Later in another
jira may be
Ya this needs a bigger change and it is not straight forward. The Hash needs a
continuous byte[] to work with. So we need to have the key part of the Cell.
If we try to operate as Cell it is going to be difficult with that algo design.
So anyway we need to do a copy - better we can have it byte[] for now.

Bloomfilter path to work with Byte buffered cells
-

Key: HBASE-14144
URL: https://issues.apache.org/jira/browse/HBASE-14144
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 2.0.0

Attachments: HBASE-14144.patch, HBASE-14144_1.patch,
HBASE-14144_2.patch, HBASE-14144_3.patch, HBASE-14144_4.patch

This JIRA is to check if there will be a need to make the bloom filters to
work with ByteBuffer cells. During POC this path created lot of duplicated
code but considering other refactorings done in this path may lead to less
duplication. This JIRA is a placeholder.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14196) Thrift server idle connection timeout issue


[ 
https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694763#comment-14694763
 ] 

Hudson commented on HBASE-14196:


SUCCESS: Integrated in HBase-1.3 #106 (See 
[https://builds.apache.org/job/HBase-1.3/106/])
HBASE-14196 Thrift server idle connection timeout issue (Vladimir Rodionov) 
(apurtell: rev e8b5e922cb1741ccc4954ef2ba0f97b5d6d39079)
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
* 
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java


 Thrift server idle connection timeout issue
 ---

 Key: HBASE-14196
 URL: https://issues.apache.org/jira/browse/HBASE-14196
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3

 Attachments: HBASE-14196.patch


 When idle connection get cleaned from Thrift server, underlying table 
 instances are still cached in a thread local cache.
 This is the antipattern. Table objects are lightweight and should not be 
 cached, besides this, underlying connections can be closed by periodic 
 connection cleaner chore (ConnectionCache) and cached table instances may 
 become invalid. This is Thrift1 specific issue. Thrift2 is OK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions


[ 
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694764#comment-14694764
 ] 

Hudson commented on HBASE-14098:


SUCCESS: Integrated in HBase-1.3 #106 (See 
[https://builds.apache.org/job/HBase-1.3/106/])
HBASE-14098 Allow dropping caches behind compactions (eclark: rev 
0dbaf0ea5145b318b18b487d8d9bc2d2eb67e19d)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/FSDataInputStreamWrapper.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeCompactor.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java


 Allow dropping caches behind compactions
 

 Key: HBASE-14098
 URL: https://issues.apache.org/jira/browse/HBASE-14098
 Project: HBase
  Issue Type: Bug
  Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 2.0.0, 1.2.0

 Attachments: HBASE-14098-v1.patch, HBASE-14098-v2.patch, 
 HBASE-14098-v3.patch, HBASE-14098-v4.patch, HBASE-14098-v5.patch, 
 HBASE-14098-v6.patch, HBASE-14098-v7-branch-1.patch, HBASE-14098-v7.patch, 
 HBASE-14098.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13985) Add configuration to skip validating HFile format when bulk loading


[ 
https://issues.apache.org/jira/browse/HBASE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694762#comment-14694762
 ] 

Hudson commented on HBASE-13985:


SUCCESS: Integrated in HBase-1.3 #106 (See 
[https://builds.apache.org/job/HBase-1.3/106/])
HBASE-13985 Add configuration to skip validating HFile format when bulk loading 
(Victor Xu) (apurtell: rev ca19f961a25dce5359bfb9b35c0bbbd64ec0fb0b)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java


 Add configuration to skip validating HFile format when bulk loading
 ---

 Key: HBASE-13985
 URL: https://issues.apache.org/jira/browse/HBASE-13985
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.98.13
Reporter: Victor Xu
Assignee: Victor Xu
Priority: Minor
  Labels: regionserver
 Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0

 Attachments: HBASE-13985-v2.patch, HBASE-13985-v3.patch, 
 HBASE-13985.patch


 When bulk loading millions of HFile into one HTable, checking HFile format is 
 the most time-consuming phase. Maybe we could use a parallel mechanism to 
 increase the speed, but when it comes to millions of HFiles, it may still 
 cost dozens of minutes. So I think it's necessary to add an option for 
 advanced user to bulkload without checking HFile format at all. 
 Of course, the default value of this option should be true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14144) Bloomfilter path to work with Byte buffered cells

[
https://issues.apache.org/jira/browse/HBASE-14144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695174#comment-14695174
]

Anoop Sam John commented on HBASE-14144:

bq.The rowBloomKey is needed here for the Deletefamily bloom cases. So the
kvKey is a cell created out of row and col and in this rowBloomKey case we only
need the row part of the Cell. Also the
Yes. Discussed with Ram here. We can see how we can better do it with out much
of the obj creations. For a later issue. Those will be much involved changes in
the Bloom area.

FirstOnRowBBFakeCell - Now also all the class names are not consistent. It
has to follow naming pattern of FakeBBCell. You changed for some.. Pls
correct all such places. Can fix on commit. +1 otherwise.

Bloomfilter path to work with Byte buffered cells
-

Attachments: HBASE-14144.patch, HBASE-14144_1.patch,
HBASE-14144_2.patch, HBASE-14144_3.patch, HBASE-14144_4.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14054) Acknowledged writes may get lost if regionserver clock is set backwards

[
https://issues.apache.org/jira/browse/HBASE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695100#comment-14695100
]

Hudson commented on HBASE-14054:

FAILURE: Integrated in HBase-1.1 #612 (See
[https://builds.apache.org/job/HBase-1.1/612/])
HBASE-14054 Acknowledged writes may get lost if regionserver clock is set
backwards (enis: rev fb30bf68bf31c73117b278aa512fad0eb4c5c79e)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java

Acknowledged writes may get lost if regionserver clock is set backwards
---

Attachments: hbase-14054_v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14189) BlockCache options should consider CF Level BlockCacheEnabled setting


[ 
https://issues.apache.org/jira/browse/HBASE-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695105#comment-14695105
 ] 

Hadoop QA commented on HBASE-14189:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12750245/HBASE-14189_v1.patch
  against master branch at commit 9c69bf766fcad024bef5760f242cae2bc609b374.
  ATTACHMENT ID: 12750245

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint

 {color:red}-1 core zombie tests{color}.  There are 2 zombie test(s):   
at 
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testWALfencing(TestMasterFailoverWithProcedures.java:216)
at 
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testWALfencingWithWALRolling(TestMasterFailoverWithProcedures.java:183)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/15083//console

This message is automatically generated.

 BlockCache options should consider CF Level BlockCacheEnabled setting
 -

 Key: HBASE-14189
 URL: https://issues.apache.org/jira/browse/HBASE-14189
 Project: HBase
  Issue Type: Improvement
  Components: BlockCache
Affects Versions: 2.0.0
Reporter: Heng Chen
Assignee: Heng Chen
 Attachments: HBASE-14189.patch, HBASE-14189_v1.patch


 While using BlockCache,  we use {{cacheDataOnRead}}({{cacheDataOnWrite}}) 
 represents for whether we should cache block after read(write) block from(to) 
 hdfs.  We should honour BC setting and CF Level cache setting while using 
 BlockCache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long

2015-08-13 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695115#comment-14695115
]

Enis Soztutar commented on HBASE-11368:
---

I was reading HBASE-4552 and RegionScannerImpl code again to try to understand
why we need the write lock for multi-CF bulk loads in the first place. It seems
that it was put there to ensure atomicity, but that should be guaranteed with
the seqId / mvcc combination and not via region write lock. However, the bulk
load files obtain a seqId, and acquiring the region write lock will block all
flushes which may be the reason. On bulk load, we call
HStore.notifyChangedReadersObservers(), which resets the KVHeap, but we never
reset the RegionScanner from my reading of code. Is this a bug? The current
scanners should not see the new bulk loaded data (via mvcc) so maybe it is ok?

Multi-column family BulkLoad fails if compactions go on too long

Key: HBASE-11368
URL: https://issues.apache.org/jira/browse/HBASE-11368
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: Qiang Tian
Attachments: hbase-11368-0.98.5.patch, hbase11368-master.patch,
key_stacktrace_hbase10882.TXT, performance_improvement_verification_98.5.patch

Compactions take a read lock. If a multi-column family region, before bulk
loading, we want to take a write lock on the region. If the compaction takes
too long, the bulk load fails.
Various recipes include:
+ Making smaller regions (lame)
+ [~victorunique] suggests major compacting just before bulk loading over in
HBASE-10882 as a work around.
Does the compaction need a read lock for that long? Does the bulk load need
a full write lock when multiple column families? Can we fail more gracefully
at least?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14054) Acknowledged writes may get lost if regionserver clock is set backwards

[
https://issues.apache.org/jira/browse/HBASE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695130#comment-14695130
]

Hudson commented on HBASE-14054:

FAILURE: Integrated in HBase-1.2 #107 (See
[https://builds.apache.org/job/HBase-1.2/107/])
HBASE-14054 Acknowledged writes may get lost if regionserver clock is set
backwards (enis: rev 59c731b24549e846e1905c43bd3e5adf121638cf)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java

Acknowledged writes may get lost if regionserver clock is set backwards
---

Attachments: hbase-14054_v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11902) RegionServer was blocked while aborting

2015-08-13 Thread Hiroshi Ikeda (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695194#comment-14695194
 ] 

Hiroshi Ikeda commented on HBASE-11902:
---

{code}
if (getValue(oldValAndFlags) == 1) return; // There were no operations 
outstanding.
synchronized (this) { this.wait(); }
{code}

If DrainBarrier#endOp calls notifyAll just before the synchronized block, this 
may wait forever.
(BTW, some of tests for DrainBarrier are also required to fix because they 
catch AssertionError thrown by JUnit.)

 RegionServer was blocked while aborting
 ---

 Key: HBASE-11902
 URL: https://issues.apache.org/jira/browse/HBASE-11902
 Project: HBase
  Issue Type: Bug
  Components: regionserver, wal
Affects Versions: 0.98.4
 Environment: hbase-0.98.4, hadoop-2.3.0-cdh5.1, jdk1.7
Reporter: Victor Xu
Assignee: Qiang Tian
 Attachments: hbase-hadoop-regionserver-hadoop461.cm6.log, 
 hbase11902-master.patch, hbase11902-master_v2.patch, 
 hbase11902-master_v3.patch, jstack_hadoop461.cm6.log


 Generally, regionserver automatically aborts when isHealth() returns false. 
 But it sometimes got blocked while aborting. I saved the jstack and logs, and 
 found out that it was caused by datanodes failures. The regionserver60020 
 thread was blocked while closing WAL. 
 This issue doesn't happen so frequently, but if it happens, it always leads 
 to huge amount of requests failure. The only way to do is KILL -9.
 I think it's a bug, but I haven't found a decent solution. Does anyone have 
 the same problem?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14214) list_labels shouldn't raise ArgumentError if no labels are defined


[ 
https://issues.apache.org/jira/browse/HBASE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695223#comment-14695223
 ] 

Hudson commented on HBASE-14214:


FAILURE: Integrated in HBase-TRUNK #6725 (See 
[https://builds.apache.org/job/HBase-TRUNK/6725/])
HBASE-14214 list_labels shouldn't raise ArgumentError if no labels are defined. 
(anoopsamjohn: rev 9c69bf766fcad024bef5760f242cae2bc609b374)
* hbase-shell/src/main/ruby/hbase/visibility_labels.rb


 list_labels shouldn't raise ArgumentError if no labels are defined 
 ---

 Key: HBASE-14214
 URL: https://issues.apache.org/jira/browse/HBASE-14214
 Project: HBase
  Issue Type: Bug
Reporter: Andrew Purtell
Assignee: Anoop Sam John
Priority: Minor
 Fix For: 2.0.0, 1.3.0

 Attachments: HBASE-14214.patch


 list_labels shouldn't raise ArgumentError if no labels are defined. The empty 
 list is a valid return case, and anyway ArgumentError should be used when 
 arguments to the command are erroneous. Here the command is well formed:
 {noformat}
 hbase(main):001:0 list_labels
 ERROR: No auth label defined
 Here is some help for this command:
 List the visibility labels defined in the system.
 Optional regular expression parameter could be used to filter the labels 
 being returned.
 Syntax : list_labels
 For example:
 hbase list_labels 'secret.*'
 hbase list_labels
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14054) Acknowledged writes may get lost if regionserver clock is set backwards

[
https://issues.apache.org/jira/browse/HBASE-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695226#comment-14695226
]

Hudson commented on HBASE-14054:

FAILURE: Integrated in HBase-1.3 #107 (See
[https://builds.apache.org/job/HBase-1.3/107/])
HBASE-14054 Acknowledged writes may get lost if regionserver clock is set
backwards (enis: rev a0086e97ad1966c2c59fde832ed60dbab9ca9e43)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java

Acknowledged writes may get lost if regionserver clock is set backwards
---

Attachments: hbase-14054_v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14214) list_labels shouldn't raise ArgumentError if no labels are defined