date:20141216


[ 
https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247988#comment-14247988
 ] 

Hadoop QA commented on HBASE-12699:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12687442/HBASE-12699.v1.master.patch
  against master branch at commit 96c6b9815ddbc9f2589655df4ad2381af04ac9f8.
  ATTACHMENT ID: 12687442

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+# DEFERRED_LOG_FLUSH is deprecated and was replaced by DURABILITY. 
 To keep backward compatible, it still exists.
+# However, it has to set before DURABILITY so that DURABILITY could 
overwrite if both args are set
+# DEFERRED_LOG_FLUSH is deprecated and was replaced by DURABILITY.  To 
keep backward compatible, it still exists.
+# However, it has to set before DURABILITY so that DURABILITY could 
overwrite if both args are set

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

 {color:red}-1 core zombie tests{color}.  There are 2 zombie test(s):   
at 
org.apache.mahout.clustering.fuzzykmeans.TestFuzzyKmeansClustering.testFuzzyKMeansMRJob(TestFuzzyKmeansClustering.java:195)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1558)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:736)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:786)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:745)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:647)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:681)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:692)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:787)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:385)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:555)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:79)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:491)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:501)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:399)
at 
org.apache.mahout.clustering.streaming.cluster.BallKMeansTest.testClusteringMultipleRuns(BallKMeansTest.java:79)

Test results:

[jira] [Commented] (HBASE-12673) Add a UT to read mob file when the mob hfile moving from the mob dir to the archive dir

2014-12-16 Thread Jiajia Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247995#comment-14247995
 ] 

Jiajia Li commented on HBASE-12673:
---

hi, [~j...@cloudera.com], as you know when hbase reads mob cell, it has two 
steps.
# Read the ref cell from the HBase, and get the cell value which is the mob 
file name.
# HBase has two possible locations to read the mob cell, one is the 
mobWorkingDir/fileName, archiveDir/fileName. When the mob file is not in the 
mobWorkingDir, HBase will try the second location.But now we only retry after 
the 
FileNotFoundException(https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L312)
Do you means a table deletion on the original table happens in the middle of 
the read operation will throw other IOExceptions? 
I don't know the HFileLink how to guarantee the case in the MOB? Can you please 
give a more detailed description? Thanks

 Add a UT to read mob file when the mob hfile moving from the mob dir to the 
 archive dir
 ---

 Key: HBASE-12673
 URL: https://issues.apache.org/jira/browse/HBASE-12673
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jiajia Li
Assignee: Jiajia Li
 Fix For: hbase-11339

 Attachments: HBASE-12673.patch


 add a unit test to scan the cloned table when deleting the original table, 
 and the steps as following:
 1) create a table with mobs, 
 2) snapshot it, 
 3) clone it as a a different table
 4) have a read workload on the snapshot
 5) delete the original table



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12636) Avoid too many write operations on zookeeper in replication

2014-12-16 Thread Liu Shaohui (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248100#comment-14248100
 ] 

Liu Shaohui commented on HBASE-12636:
-

[~lhofhansl] [~stack]
Repeated replication data will happen in many cases in current codebase.
- The network problem which make master cluster did not get response of 
replication but the replication data has been written into in slave cluster.
- A moving region in salve cluster which make the replication failed, but part 
of replication data has been into in slave cluster.

This patch just make repeated replication data more frequently.

We may open another issue to check if the replication operation is idempotent?

 Avoid too many write operations on zookeeper in replication
 ---

 Key: HBASE-12636
 URL: https://issues.apache.org/jira/browse/HBASE-12636
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.11
Reporter: Liu Shaohui
Assignee: Liu Shaohui
  Labels: replication
 Fix For: 1.0.0

 Attachments: HBASE-12635-v2.diff, HBASE-12636-v1.diff


 In our production cluster, we found there are about over 1k write operations 
 per second on zookeeper from hbase replication. The reason is that the 
 replication source will write the log position to zookeeper for every edit 
 shipping. If the current replicating WAL is just the WAL that regionserver is 
 writing to,  each skipping will be very small but the frequency is very high, 
 which causes many write operations on zookeeper.
 A simple solution is that writing log position to zookeeper when position 
 diff or skipped edit number is larger than a threshold, not every  edit 
 shipping.
 Suggestions are welcomed, thx~



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow


 [ 
https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-12223:
---
Status: Patch Available  (was: Open)

 MultiTableInputFormatBase.getSplits is too slow
 ---

 Key: HBASE-12223
 URL: https://issues.apache.org/jira/browse/HBASE-12223
 Project: HBase
  Issue Type: Improvement
  Components: Client
Affects Versions: 0.94.15
Reporter: shanwen
Assignee: YuanBo Peng
Priority: Minor
 Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27

 Attachments: HBASE-12223-v1.patch, HBASE-12223.patch


 when use Multiple scan,getSplits is too slow,800 scans take five minutes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow

[
https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248155#comment-14248155
]

Hadoop QA commented on HBASE-12223:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12687450/HBASE-12223-v1.patch
against master branch at commit 96c6b9815ddbc9f2589655df4ad2381af04ac9f8.
ATTACHMENT ID: 12687450

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12095//console

This message is automatically generated.

MultiTableInputFormatBase.getSplits is too slow
---

Key: HBASE-12223
URL: https://issues.apache.org/jira/browse/HBASE-12223
Project: HBase
Issue Type: Improvement
Components: Client
Affects Versions: 0.94.15
Reporter: shanwen
Assignee: YuanBo Peng
Priority: Minor
Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27

Attachments: HBASE-12223-v1.patch, HBASE-12223.patch

when use Multiple scan,getSplits is too slow,800 scans take five minutes

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12641) Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster

2014-12-16 Thread Liu Shaohui (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-12641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248192#comment-14248192
]

Liu Shaohui commented on HBASE-12641:
-

[~apurtell]
{quote}
Why the 'if (!node.startsWith(zkw.baseZNode))' shortcut?
{quote}
See HBASE-7258: HBase will create the baseZNode recursively if the parent node
does not exist.
if zookeeper.znode.parent is /service/hbase/, we don't want set acl on node
/service when hbase creates this node.
So we add this shortcut.

Grant all permissions of hbase zookeeper node to hbase superuser in a secure
cluster

Key: HBASE-12641
URL: https://issues.apache.org/jira/browse/HBASE-12641
Project: HBase
Issue Type: Improvement
Components: Zookeeper
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Fix For: 1.0.0

Attachments: HBASE-12641-v1.diff

Currently in a secure cluster, only the master/regionserver kerberos user can
manage the znode of hbase. But he master/regionserver kerberos user is for
rpc connection and we usually use another super user to manage the cluster.
In some special scenarios, we need to manage the data of znode with the
supper user.
eg:
a, To get the data of the znode for debugging.
b, HBASE-8253: We need to delete the znode for the corrupted hlog to avoid it
block the replication.
So we grant all permissions of hbase zookeeper node to hbase superuser during
creating these znodes.
Suggestions are welcomed.
[~apurtell]

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job

2014-12-16 Thread Weichen Ye (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Weichen Ye updated HBASE-12590:
---
Attachment: HBASE-12590-v3.patch

[~j...@cloudera.com]
Hi, would you please take a look at this new patch?

in the new patch:
1, re-design the function for getting split point in large region
2, add a new mode for binary keys. The default mode is for text keys. User can
swith by setting a new configuration: hbase.table.row.textkey
3, add new tests for both text keys and binary keys

A solution for data skew in HBase-Mapreduce Job
---

Key: HBASE-12590
URL: https://issues.apache.org/jira/browse/HBASE-12590
Project: HBase
Issue Type: Improvement
Components: mapreduce
Reporter: Weichen Ye
Attachments: A Solution for Data Skew in HBase-MapReduce Job
(Version2).pdf, HBASE-12590-v3.patch, HBase-12590-v1.patch,
HBase-12590-v2.patch

1, Motivation
In production environment, data skew is a very common case. A HBase table
always contains a lot of small regions and several large regions. Small
regions waste a lot of computing resources. If we use a job to scan a table
with 3000 small regions, we need a job with 3000 mappers. Large regions
always block the job. If in a 100-region table, one region is far larger then
the other 99 regions. When we run a job with the table as input, 99 mappers
will be completed very quickly, and we need to wait for the last mapper for a
long time.
2, Configuration
Add two new configuration.
hbase.mapreduce.split.autobalance = true means enabling the “auto balance” in
HBase-MapReduce jobs. The default value is false.
hbase.mapreduce.split.targetsize = 1073741824 (default 1GB). The target size
of mapreduce splits.
If a region size is large than the target size, cut the region into two
split.If the sum of several small continuous region size less than the target
size, combine these regions into one split.
Example:
In attachment
Welcome to the Review Board.
https://reviews.apache.org/r/28494/diff/#

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job

2014-12-16 Thread Weichen Ye (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Weichen Ye updated HBASE-12590:
---
Attachment: A Solution for Data Skew in HBase-MapReduce Job (Version3).pdf

A solution for data skew in HBase-Mapreduce Job
---

Key: HBASE-12590
URL: https://issues.apache.org/jira/browse/HBASE-12590
Project: HBase
Issue Type: Improvement
Components: mapreduce
Reporter: Weichen Ye
Attachments: A Solution for Data Skew in HBase-MapReduce Job
(Version2).pdf, A Solution for Data Skew in HBase-MapReduce Job
(Version3).pdf, HBASE-12590-v3.patch, HBase-12590-v1.patch,
HBase-12590-v2.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job

2014-12-16 Thread Weichen Ye (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248199#comment-14248199
]

Weichen Ye commented on HBASE-12590:

Latest diff on review board: https://reviews.apache.org/r/28494/diff/

A solution for data skew in HBase-Mapreduce Job
---

Key: HBASE-12590
URL: https://issues.apache.org/jira/browse/HBASE-12590
Project: HBase
Issue Type: Improvement
Components: mapreduce
Reporter: Weichen Ye
Attachments: A Solution for Data Skew in HBase-MapReduce Job
(Version2).pdf, A Solution for Data Skew in HBase-MapReduce Job
(Version3).pdf, HBASE-12590-v3.patch, HBase-12590-v1.patch,
HBase-12590-v2.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job

[
https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248276#comment-14248276
]

Hadoop QA commented on HBASE-12590:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12687480/HBASE-12590-v3.patch
against master branch at commit 96c6b9815ddbc9f2589655df4ad2381af04ac9f8.
ATTACHMENT ID: 12687480

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:red}-1 checkstyle{color}. The applied patch generated
2091 checkstyle errors (more than the master's current 2089 errors).

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//artifact/patchprocess/checkstyle-aggregate.html

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12096//console

This message is automatically generated.

A solution for data skew in HBase-Mapreduce Job
---

Key: HBASE-12590
URL: https://issues.apache.org/jira/browse/HBASE-12590
Project: HBase
Issue Type: Improvement
Components: mapreduce
Reporter: Weichen Ye
Attachments: A Solution for Data Skew in HBase-MapReduce Job
(Version2).pdf, A Solution for Data Skew in HBase-MapReduce Job
(Version3).pdf, HBASE-12590-v3.patch, HBase-12590-v1.patch,
HBase-12590-v2.patch

[jira] [Commented] (HBASE-12673) Add a UT to read mob file when the mob hfile moving from the mob dir to the archive dir


[ 
https://issues.apache.org/jira/browse/HBASE-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248323#comment-14248323
 ] 

Jonathan Hsieh commented on HBASE-12673:


Thanks for the pointer in HMobStore @ 312.  The retry there should address the 
situation I was concerned about.  Looking at that code again one other concern 
comes up -- do you know if line 319 in there will throw another exception if we 
got the FNFE on line 313?  (we'd have an open file instance that got moved --- 
not sure what would happen on close on line 319).  Can we change it so that we 
capture the other exceptions that could be caught there [1] and then try the 
next location?

HFileLink essentially makes this file redirection mechanism transparent and 
would potentially make the code easier to follow.If you can convince me 
that the line 319 concerns isn't a problem with a unit tests I'll drop the 
HFileLink request.  As it stands it might be a little difficult to mock it out 
but would be helpful.

[1] 
https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java#L124

 Add a UT to read mob file when the mob hfile moving from the mob dir to the 
 archive dir
 ---

 Key: HBASE-12673
 URL: https://issues.apache.org/jira/browse/HBASE-12673
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jiajia Li
Assignee: Jiajia Li
 Fix For: hbase-11339

 Attachments: HBASE-12673.patch


 add a unit test to scan the cloned table when deleting the original table, 
 and the steps as following:
 1) create a table with mobs, 
 2) snapshot it, 
 3) clone it as a a different table
 4) have a read workload on the snapshot
 5) delete the original table



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-12673) Add a UT to read mob file when the mob hfile moving from the mob dir to the archive dir


[ 
https://issues.apache.org/jira/browse/HBASE-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248323#comment-14248323
 ] 

Jonathan Hsieh edited comment on HBASE-12673 at 12/16/14 2:43 PM:
--

Thanks for the pointer in HMobStore @ 312.  The retry there should address the 
situation I was concerned about.  Looking at that code again one other concern 
comes up -- do you know if line 319 in there will throw another exception if we 
got the FNFE on line 313?  (we'd have an open file instance that got moved --- 
not sure what would happen on close on line 319).  Can we change it so that we 
capture the other exceptions that could be caught there [1] and then try the 
next location?

HFileLink essentially makes this file redirection mechanism transparent and 
would potentially make the code easier to follow.  I'd prefer it if we could 
use that code so if we find other cases we can just fix it in one centralized 
place. 

[1] 
https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java#L124


was (Author: jmhsieh):
Thanks for the pointer in HMobStore @ 312.  The retry there should address the 
situation I was concerned about.  Looking at that code again one other concern 
comes up -- do you know if line 319 in there will throw another exception if we 
got the FNFE on line 313?  (we'd have an open file instance that got moved --- 
not sure what would happen on close on line 319).  Can we change it so that we 
capture the other exceptions that could be caught there [1] and then try the 
next location?

HFileLink essentially makes this file redirection mechanism transparent and 
would potentially make the code easier to follow.If you can convince me 
that the line 319 concerns isn't a problem with a unit tests I'll drop the 
HFileLink request.  As it stands it might be a little difficult to mock it out 
but would be helpful.

[1] 
https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java#L124

 Add a UT to read mob file when the mob hfile moving from the mob dir to the 
 archive dir
 ---

 Key: HBASE-12673
 URL: https://issues.apache.org/jira/browse/HBASE-12673
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jiajia Li
Assignee: Jiajia Li
 Fix For: hbase-11339

 Attachments: HBASE-12673.patch


 add a unit test to scan the cloned table when deleting the original table, 
 and the steps as following:
 1) create a table with mobs, 
 2) snapshot it, 
 3) clone it as a a different table
 4) have a read workload on the snapshot
 5) delete the original table



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12629) Remove hbase.regionsizecalculator.enable from RegionSizeCalculator


[ 
https://issues.apache.org/jira/browse/HBASE-12629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248376#comment-14248376
 ] 

Jonathan Hsieh commented on HBASE-12629:


[~enis] ping.

 Remove hbase.regionsizecalculator.enable from RegionSizeCalculator
 --

 Key: HBASE-12629
 URL: https://issues.apache.org/jira/browse/HBASE-12629
 Project: HBase
  Issue Type: Improvement
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 1.0.0, 2.0.0

 Attachments: 
 0001-HBASE-12629-Remove-hbase.regionsizecalculator.enable.patch


 The RegionSizeCalculator has a option to disable it.  It is on by default and 
 end-to-end use with it disabled is not tested or used anywhere except for a 
 simple unit test.  This removes it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception throw when setting DEFERRED_LOG_FLUSH=true


[ 
https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248391#comment-14248391
 ] 

stack commented on HBASE-12699:
---

+1 Failure not related.

 undefined method `setAsyncLogFlush' exception throw when setting 
 DEFERRED_LOG_FLUSH=true
 -

 Key: HBASE-12699
 URL: https://issues.apache.org/jira/browse/HBASE-12699
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 2.0.0, 0.99.2
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
 Attachments: HBASE-12699.v1.master.patch

   Original Estimate: 24h
  Remaining Estimate: 24h

 In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, 
 an undefined method `setAsyncLogFlush' exception was thrown.  
 This is due to that DEFERRED_LOG_FLUSH was deprecated and the 
 setAsyncLogFlush method was removed.  It was replaced by DURABILITY.
 DEFERRED_LOG_FLUSH=true is the same as DURABILITY='ASYNC_WAL'
 The default is DURABILITY='SYNC_WAL', which is the same as the default 
 DEFERRED_LOG_FLUSH=false
 We should ask user to use the DURABILITY setting.  In the meantime, for 
 backward compatibility, the hbase shell should still allow setting 
 DEFERRED_LOG_FLUSH.  Internally, instead of calling setAsyncLogFlush, it 
 should call setDurability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11861) Native MOB Compaction mechanisms.

[
https://issues.apache.org/jira/browse/HBASE-11861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248395#comment-14248395
]

Jonathan Hsieh commented on HBASE-11861:

bq. This is why I insist to run the mob compaction in regions. If we do the mob
compaction out of region or across regions, we have to locks the major
compactions globally.

nice catch on that race condition -- I buy it. This is essentially the same as
with the MR sweeper approach right?

So we'd need to guarantee that the compacted mob and the bulkload of the new
references block a major compaction on the region that the ref bulk load is
happening on. This means no major compactions before step #2, but allowed
after step #4.

Let's spell out the costs of the different approaches. -- the del mob global
scan for the mob compaction approach and the per region mob compaction.

Meanwhile I noticed you file a new jira for counts and I filed one for the del
mob generator. We can get code started on those, and hash out this higher
level design while doing so.

bq. I think we could leave the expired(live longer than TTL) cells out of the
del files. Let the ExpiredMobFileCleaner to handle those mob files directly.

sounds reasonable. We need to enforce the mob file time ordering though to
make sure the mob compaction is effective.

Native MOB Compaction mechanisms.
-

Key: HBASE-11861
URL: https://issues.apache.org/jira/browse/HBASE-11861
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: 2.0.0
Reporter: Jonathan Hsieh
Attachments: 141030-mob-compaction.pdf, mob compaction.pdf

Currently, the first cut of mob will have external processes to age off old
mob data (the ttl cleaner), and to compact away deleted or over written data
(the sweep tool).
From an operational point of view, having two external tools, especially one
that relies on MapReduce is undesirable. In this issue we'll tackle
integrating these into hbase without requiring external processes.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12699) undefined method `setAsyncLogFlush' exception throw when setting DEFERRED_LOG_FLUSH=true

2014-12-16 Thread Stephen Yuan Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-12699:
---
Attachment: HBASE-12699.v1.branch-1.patch

 undefined method `setAsyncLogFlush' exception throw when setting 
 DEFERRED_LOG_FLUSH=true
 -

 Key: HBASE-12699
 URL: https://issues.apache.org/jira/browse/HBASE-12699
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 2.0.0, 0.99.2
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
 Attachments: HBASE-12699.v1.branch-1.patch, 
 HBASE-12699.v1.master.patch

   Original Estimate: 24h
  Remaining Estimate: 24h

 In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, 
 an undefined method `setAsyncLogFlush' exception was thrown.  
 This is due to that DEFERRED_LOG_FLUSH was deprecated and the 
 setAsyncLogFlush method was removed.  It was replaced by DURABILITY.
 DEFERRED_LOG_FLUSH=true is the same as DURABILITY='ASYNC_WAL'
 The default is DURABILITY='SYNC_WAL', which is the same as the default 
 DEFERRED_LOG_FLUSH=false
 We should ask user to use the DURABILITY setting.  In the meantime, for 
 backward compatibility, the hbase shell should still allow setting 
 DEFERRED_LOG_FLUSH.  Internally, instead of calling setAsyncLogFlush, it 
 should call setDurability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work logged] (HBASE-12699) undefined method `setAsyncLogFlush' exception throw when setting DEFERRED_LOG_FLUSH=true

2014-12-16 Thread Stephen Yuan Jiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12699?focusedWorklogId=18771page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-18771
 ]

Stephen Yuan Jiang logged work on HBASE-12699:
--

Author: Stephen Yuan Jiang
Created on: 16/Dec/14 17:28
Start Date: 16/Dec/14 17:27
Worklog Time Spent: 4h 

Issue Time Tracking
---

Worklog Id: (was: 18771)
Time Spent: 4h
Remaining Estimate: 1h  (was: 24h)

 undefined method `setAsyncLogFlush' exception throw when setting 
 DEFERRED_LOG_FLUSH=true
 -

 Key: HBASE-12699
 URL: https://issues.apache.org/jira/browse/HBASE-12699
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 2.0.0, 0.99.2
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
 Attachments: HBASE-12699.v1.branch-1.patch, 
 HBASE-12699.v1.master.patch

   Original Estimate: 24h
  Time Spent: 4h
  Remaining Estimate: 1h

 In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, 
 an undefined method `setAsyncLogFlush' exception was thrown.  
 This is due to that DEFERRED_LOG_FLUSH was deprecated and the 
 setAsyncLogFlush method was removed.  It was replaced by DURABILITY.
 DEFERRED_LOG_FLUSH=true is the same as DURABILITY='ASYNC_WAL'
 The default is DURABILITY='SYNC_WAL', which is the same as the default 
 DEFERRED_LOG_FLUSH=false
 We should ask user to use the DURABILITY setting.  In the meantime, for 
 backward compatibility, the hbase shell should still allow setting 
 DEFERRED_LOG_FLUSH.  Internally, instead of calling setAsyncLogFlush, it 
 should call setDurability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=true


 [ 
https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-12699:
---
Summary: undefined method `setAsyncLogFlush' exception thrown when setting 
DEFERRED_LOG_FLUSH=true  (was: undefined method `setAsyncLogFlush' exception 
throw when setting DEFERRED_LOG_FLUSH=true)

 undefined method `setAsyncLogFlush' exception thrown when setting 
 DEFERRED_LOG_FLUSH=true
 --

 Key: HBASE-12699
 URL: https://issues.apache.org/jira/browse/HBASE-12699
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 2.0.0, 0.99.2
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
 Attachments: HBASE-12699.v1.branch-1.patch, 
 HBASE-12699.v1.master.patch

   Original Estimate: 24h
  Time Spent: 4h
  Remaining Estimate: 1h

 In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, 
 an undefined method `setAsyncLogFlush' exception was thrown.  
 This is due to that DEFERRED_LOG_FLUSH was deprecated and the 
 setAsyncLogFlush method was removed.  It was replaced by DURABILITY.
 DEFERRED_LOG_FLUSH=true is the same as DURABILITY='ASYNC_WAL'
 The default is DURABILITY='SYNC_WAL', which is the same as the default 
 DEFERRED_LOG_FLUSH=false
 We should ask user to use the DURABILITY setting.  In the meantime, for 
 backward compatibility, the hbase shell should still allow setting 
 DEFERRED_LOG_FLUSH.  Internally, instead of calling setAsyncLogFlush, it 
 should call setDurability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=true


 [ 
https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-12699:
---
   Resolution: Fixed
Fix Version/s: 2.0.0
   1.0.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks for the patch, Stephen.

Thanks for the review, Stack.

 undefined method `setAsyncLogFlush' exception thrown when setting 
 DEFERRED_LOG_FLUSH=true
 --

 Key: HBASE-12699
 URL: https://issues.apache.org/jira/browse/HBASE-12699
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 2.0.0, 0.99.2
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
 Fix For: 1.0.0, 2.0.0

 Attachments: HBASE-12699.v1.branch-1.patch, 
 HBASE-12699.v1.master.patch

   Original Estimate: 24h
  Time Spent: 4h
  Remaining Estimate: 1h

 In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, 
 an undefined method `setAsyncLogFlush' exception was thrown.  
 This is due to that DEFERRED_LOG_FLUSH was deprecated and the 
 setAsyncLogFlush method was removed.  It was replaced by DURABILITY.
 DEFERRED_LOG_FLUSH=true is the same as DURABILITY='ASYNC_WAL'
 The default is DURABILITY='SYNC_WAL', which is the same as the default 
 DEFERRED_LOG_FLUSH=false
 We should ask user to use the DURABILITY setting.  In the meantime, for 
 backward compatibility, the hbase shell should still allow setting 
 DEFERRED_LOG_FLUSH.  Internally, instead of calling setAsyncLogFlush, it 
 should call setDurability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-5699) Run with 1 WAL in HRegionServer

2014-12-16 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-5699:
---
Attachment: 
HBASE-5699_disabled_and_regular_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff

Attaching a plot that includes running the same tests with the delegate wals as 
DisabledWALProvider, named 
HBASE-5699_disabled_and_regular_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.

This should show the limit from context switching and such in the test itself. 
The DisabledWALProvider doesn't include any of the overhead from the ringbuffer 
or sync grouping.

There are very few data points for the new test cases, so I didn't include any 
stddev bars. I just used the average for hte whole run. All of them were so 
short that they probably didn't have time to get into steady state.

(The disabled wals are at the top, the previous runs with datanode writes are 
at the bottom)

 Run with  1 WAL in HRegionServer
 -

 Key: HBASE-5699
 URL: https://issues.apache.org/jira/browse/HBASE-5699
 Project: HBase
  Issue Type: Improvement
  Components: Performance, wal
Reporter: binlijin
Assignee: Sean Busbey
Priority: Critical
 Attachments: HBASE-5699.3.patch.txt, HBASE-5699.4.patch.txt, 
 HBASE-5699_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff, 
 HBASE-5699_disabled_and_regular_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff,
  HBASE-5699_write_iops_multiwal-1_1_to_200_threads.tiff, 
 HBASE-5699_write_iops_multiwal-2_10,50,120,190,260,330,400_threads.tiff, 
 HBASE-5699_write_iops_multiwal-4_10,50,120,190,260,330,400_threads.tiff, 
 HBASE-5699_write_iops_multiwal-6_10,50,120,190,260,330,400_threads.tiff, 
 HBASE-5699_write_iops_upstream_1_to_200_threads.tiff, PerfHbase.txt, 
 hbase-5699_multiwal_400-threads_stats_sync_heavy.txt, 
 hbase-5699_total_throughput_sync_heavy.txt, 
 results-hbase5699-upstream.txt.bz2, results-hbase5699-wals-1.txt.bz2, 
 results-updated-hbase5699-wals-2.txt.bz2, 
 results-updated-hbase5699-wals-4.txt.bz2, 
 results-updated-hbase5699-wals-6.txt.bz2






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir

[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248582#comment-14248582
]

Varun Saxena commented on HBASE-12645:
--

[~stack], kindly review. The test case failure is unrelated and tests are
passing in my local setup.

bq. With this in place, after the test suite completes, we are not writing to
user homedir any more?
I removed all references of {{getHomeDir())) from code and nothing existed in
home dir after test suite completed.
Moreover, I checked all the test logs to find if some directory was created
inside home directory or not and could not find anything. So in all probability
nothing should have been created in home dir.

bq. Also, the new flag is never doc'd. What is it supposed to do? (I'm not
clear). Flag is createRootDirIfExists This means, create root dir if it exists?
But we don't check existance when we use it. Should we?
Flag name has been changed. The flag basically meant whether to fetch a new
root dir path if one already exists(has been fetched earlier for the test
class). First a path is created and then directory is created. The directory is
anyways newly created(overwritten if already there).

bq. nit: Usually the following is written as if(!createIfExists) rather than as
if (false == createIfExists) {
Made the necessary change.

bq. Would we not want this flag always set? Or some tests need it not set?
Some cases fail if we always set the flag.

HBaseTestingUtility is using ${$HOME} for rootDir
-

Key: HBASE-12645
URL: https://issues.apache.org/jira/browse/HBASE-12645
Project: HBase
Issue Type: Test
Components: test
Affects Versions: 1.0.0
Reporter: Nick Dimiduk
Assignee: Varun Saxena
Priority: Critical
Fix For: 1.0.0, 2.0.0

Attachments: HBASE-12645.002.patch, HBASE-12645.003.patch,
HBASE-12645.004.patch, HBASE-12645.004.patch, HBASE-12645.005.patch,
HBASE-12645.006.patch, HBASE-12645.patch

I noticed this while running tests on branch-1
{noformat}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec
FAILURE! - in
org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
elapsed: 0.009 sec ERROR!
java.io.FileNotFoundException: Destination exists and is not a directory:
/homes/hortonnd/hbase
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
at
org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
at
org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
{noformat}
Either the testing utility has a regression or there's a config regression in
this test.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir


 [ 
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-12645:
--
Attachment: HBASE-12645.006.patch

Retry

 HBaseTestingUtility is using ${$HOME} for rootDir
 -

 Key: HBASE-12645
 URL: https://issues.apache.org/jira/browse/HBASE-12645
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 1.0.0
Reporter: Nick Dimiduk
Assignee: Varun Saxena
Priority: Critical
 Fix For: 1.0.0, 2.0.0

 Attachments: HBASE-12645.002.patch, HBASE-12645.003.patch, 
 HBASE-12645.004.patch, HBASE-12645.004.patch, HBASE-12645.005.patch, 
 HBASE-12645.006.patch, HBASE-12645.006.patch, HBASE-12645.patch


 I noticed this while running tests on branch-1
 {noformat}
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec  
 FAILURE! - in 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits  Time 
 elapsed: 0.009 sec   ERROR!
 java.io.FileNotFoundException: Destination exists and is not a directory: 
 /homes/hortonnd/hbase
 at 
 org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
 at 
 org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
 at 
 org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
 at 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
 {noformat}
 Either the testing utility has a regression or there's a config regression in 
 this test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir


[ 
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248637#comment-14248637
 ] 

stack commented on HBASE-12645:
---

Patch looks good.

I'm not sure I understand what this means:

Whether to get a new root or data dir path even if such a path has been 
fetched earlier is decided based on flag getNewRootDirPathIfExists

What is 'get a new root or data dir path' and what does 'fetched' mean here?  
Does it mean created (create with override)? (I think it means this latter -- 
one of your javadoc param notes says so... I'd think all mention of this flag 
in javadoc would have same text?)

Usually too the above comment is put on the javadoc param '@param 
getNewRootDirPathIfExists Whether to get a '  I see you do it sometimes but 
not always.

getNewRootDirPathIfExists is not a good name for a flag.  It is the name of a 
getter method for a flag named newRootDirPathIfExists

Otherwise, the patch is great and thank you for checking we no longer write the 
root dir. I ran the patch again against hadoopqa so we can see test failure is 
not related.  T hanks.

 HBaseTestingUtility is using ${$HOME} for rootDir
 -

 Key: HBASE-12645
 URL: https://issues.apache.org/jira/browse/HBASE-12645
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 1.0.0
Reporter: Nick Dimiduk
Assignee: Varun Saxena
Priority: Critical
 Fix For: 1.0.0, 2.0.0

 Attachments: HBASE-12645.002.patch, HBASE-12645.003.patch, 
 HBASE-12645.004.patch, HBASE-12645.004.patch, HBASE-12645.005.patch, 
 HBASE-12645.006.patch, HBASE-12645.006.patch, HBASE-12645.patch


 I noticed this while running tests on branch-1
 {noformat}
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec  
 FAILURE! - in 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits  Time 
 elapsed: 0.009 sec   ERROR!
 java.io.FileNotFoundException: Destination exists and is not a directory: 
 /homes/hortonnd/hbase
 at 
 org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
 at 
 org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
 at 
 org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
 at 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
 {noformat}
 Either the testing utility has a regression or there's a config regression in 
 this test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5699) Run with 1 WAL in HRegionServer


[ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248660#comment-14248660
 ] 

stack commented on HBASE-5699:
--

Thanks [~busbey] Looks like writing datanodes puts a bit of friction on our 
write path.  I wonder how much the ringbuffer+grouping is costing us? Looking 
at the graph, you'd think adding extra WALs would make a bigger difference 
given the large gap between no-friction and one WAL. Good stuff.

 Run with  1 WAL in HRegionServer
 -

 Key: HBASE-5699
 URL: https://issues.apache.org/jira/browse/HBASE-5699
 Project: HBase
  Issue Type: Improvement
  Components: Performance, wal
Reporter: binlijin
Assignee: Sean Busbey
Priority: Critical
 Attachments: HBASE-5699.3.patch.txt, HBASE-5699.4.patch.txt, 
 HBASE-5699_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff, 
 HBASE-5699_disabled_and_regular_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff,
  HBASE-5699_write_iops_multiwal-1_1_to_200_threads.tiff, 
 HBASE-5699_write_iops_multiwal-2_10,50,120,190,260,330,400_threads.tiff, 
 HBASE-5699_write_iops_multiwal-4_10,50,120,190,260,330,400_threads.tiff, 
 HBASE-5699_write_iops_multiwal-6_10,50,120,190,260,330,400_threads.tiff, 
 HBASE-5699_write_iops_upstream_1_to_200_threads.tiff, PerfHbase.txt, 
 hbase-5699_multiwal_400-threads_stats_sync_heavy.txt, 
 hbase-5699_total_throughput_sync_heavy.txt, 
 results-hbase5699-upstream.txt.bz2, results-hbase5699-wals-1.txt.bz2, 
 results-updated-hbase5699-wals-2.txt.bz2, 
 results-updated-hbase5699-wals-4.txt.bz2, 
 results-updated-hbase5699-wals-6.txt.bz2






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir


[ 
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248687#comment-14248687
 ] 

Varun Saxena commented on HBASE-12645:
--

bq. Whether to get a new root or data dir path even if such a path has been 
fetched earlier is decided based on flag getNewRootDirPathIfExists
bq. What is 'get a new root or data dir path' and what does 'fetched' mean 
here? Does it mean created (create with override)? (I think it means this 
latter – one of your javadoc param notes says so... I'd think all mention of 
this flag in javadoc would have same text?)
Well, the creation of root directory and fetching of path of root directory are 
2 distinct operations. {{createRootDir()}} fetches the path of root directory 
by calling {{getDefaultRootDirPath()}} and then creates it(overwrites if path 
exists). 
Whenever we fetch or get a root or data directory path we mark it(variable 
{{dataTestDirOnTestFS}} will be null if path hasnt been retrieved earlier). 
The flag {{getNewRootDirPathIfExists}} indicates whether this flow of getting 
path has been hit before or not by checking whether {{dataTestDirOnTestFS}} is 
null or not.
Flag name has been kept like this for the lack of a proper name. Didnt sound 
great even to me. Probably you can suggest a better name.


 HBaseTestingUtility is using ${$HOME} for rootDir
 -

 Key: HBASE-12645
 URL: https://issues.apache.org/jira/browse/HBASE-12645
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 1.0.0
Reporter: Nick Dimiduk
Assignee: Varun Saxena
Priority: Critical
 Fix For: 1.0.0, 2.0.0

 Attachments: HBASE-12645.002.patch, HBASE-12645.003.patch, 
 HBASE-12645.004.patch, HBASE-12645.004.patch, HBASE-12645.005.patch, 
 HBASE-12645.006.patch, HBASE-12645.006.patch, HBASE-12645.patch


 I noticed this while running tests on branch-1
 {noformat}
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec  
 FAILURE! - in 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits  Time 
 elapsed: 0.009 sec   ERROR!
 java.io.FileNotFoundException: Destination exists and is not a directory: 
 /homes/hortonnd/hbase
 at 
 org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
 at 
 org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
 at 
 org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
 at 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
 {noformat}
 Either the testing utility has a regression or there's a config regression in 
 this test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=true

2014-12-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248686#comment-14248686
 ] 

Hudson commented on HBASE-12699:


SUCCESS: Integrated in HBase-1.0 #588 (See 
[https://builds.apache.org/job/HBase-1.0/588/])
HBASE-12699 undefined method 'setAsyncLogFlush' exception thrown when setting 
DEFERRED_LOG_FLUSH=true (Stephen Jiang) (tedyu: rev 
a9645e3e97ea06fa0acf277680124535226b4297)
* hbase-shell/src/main/ruby/hbase/admin.rb


 undefined method `setAsyncLogFlush' exception thrown when setting 
 DEFERRED_LOG_FLUSH=true
 --

 Key: HBASE-12699
 URL: https://issues.apache.org/jira/browse/HBASE-12699
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 2.0.0, 0.99.2
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
 Fix For: 1.0.0, 2.0.0

 Attachments: HBASE-12699.v1.branch-1.patch, 
 HBASE-12699.v1.master.patch

   Original Estimate: 24h
  Time Spent: 4h
  Remaining Estimate: 1h

 In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, 
 an undefined method `setAsyncLogFlush' exception was thrown.  
 This is due to that DEFERRED_LOG_FLUSH was deprecated and the 
 setAsyncLogFlush method was removed.  It was replaced by DURABILITY.
 DEFERRED_LOG_FLUSH=true is the same as DURABILITY='ASYNC_WAL'
 The default is DURABILITY='SYNC_WAL', which is the same as the default 
 DEFERRED_LOG_FLUSH=false
 We should ask user to use the DURABILITY setting.  In the meantime, for 
 backward compatibility, the hbase shell should still allow setting 
 DEFERRED_LOG_FLUSH.  Internally, instead of calling setAsyncLogFlush, it 
 should call setDurability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir


[ 
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248692#comment-14248692
 ] 

Varun Saxena commented on HBASE-12645:
--

Correcting myself.
The flag {{getNewRootDirPathIfExists}} indicates whether to get a new root 
directory path if this flow of getting path has been hit before (whether path 
has been hit or not is done by checking whether dataTestDirOnTestFS is null or 
not)

 HBaseTestingUtility is using ${$HOME} for rootDir
 -

 Key: HBASE-12645
 URL: https://issues.apache.org/jira/browse/HBASE-12645
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 1.0.0
Reporter: Nick Dimiduk
Assignee: Varun Saxena
Priority: Critical
 Fix For: 1.0.0, 2.0.0

 Attachments: HBASE-12645.002.patch, HBASE-12645.003.patch, 
 HBASE-12645.004.patch, HBASE-12645.004.patch, HBASE-12645.005.patch, 
 HBASE-12645.006.patch, HBASE-12645.006.patch, HBASE-12645.patch


 I noticed this while running tests on branch-1
 {noformat}
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec  
 FAILURE! - in 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits  Time 
 elapsed: 0.009 sec   ERROR!
 java.io.FileNotFoundException: Destination exists and is not a directory: 
 /homes/hortonnd/hbase
 at 
 org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
 at 
 org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
 at 
 org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
 at 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
 {noformat}
 Either the testing utility has a regression or there's a config regression in 
 this test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12644) Visibility Labels: issue with storing super users in labels table

2014-12-16 Thread Jerry He (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248697#comment-14248697
]

Jerry He commented on HBASE-12644:
--

Hi, [~tedyu], [~anoop.hbase]

I do not find an option in the menu for me to add release note. Probably I do
not have the permission.
Here is the info that needs to be added. Feel free to change the wording as
you desire. Thanks.

-
The system visibility label authorization for super users will no longer be
persisted in hbase:labels table. Super users will be determined at server
startup time. They will have all the permissions for Visibility labels.
If you have a prior deployment that had super users' system label persisted in
hbase:labels, you can clean up by invoking the shell command 'clear_auths'.
For example: clear_auths 'old_superuser', 'system'
This is particular necessary when you change super users, i.e. a previous super
user is no longer a super user.

Visibility Labels: issue with storing super users in labels table
-

Key: HBASE-12644
URL: https://issues.apache.org/jira/browse/HBASE-12644
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.98.8, 0.99.2
Reporter: Jerry He
Assignee: Jerry He
Fix For: 1.0.0, 2.0.0

Attachments: 12644-0.98.patch, HBASE-12644-master-v2.patch,
HBASE-12644-master-v3.patch, HBASE-12644-master.patch

Super users have all the permissions for ACL and Visibility labels.
They are defined in hbase-site.xml.
Currently in VisibilityController, we persist super user with their system
permission in hbase:labels.
This makes change in super user difficult.
There are two issues:
In the current DefaultVisibilityLabelServiceImpl.addSystemLabel, we only add
super user when we initially create the 'system' label.
No additional update after that even if super user changed. See code for
details.

Additionally, there is no mechanism to remove any super user from the labels
table.

We probably should not persist super users in the labels table.
They are in hbase-site.xml and can just stay in labelsCache and used from
labelsCache after retrieval by Visibility Controller.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12644) Visibility Labels: issue with storing super users in labels table

[
https://issues.apache.org/jira/browse/HBASE-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ted Yu updated HBASE-12644:
---
Release Note:
The system visibility label authorization for super users will no longer be
persisted in hbase:labels table. Super users will be determined at server
startup time. They will have all the permissions for Visibility labels.
If you have a prior deployment that had super users' system label persisted in
hbase:labels, you can clean up by invoking the shell command 'clear_auths'.
For example: clear_auths 'old_superuser', 'system'
This is particularly necessary when you change super users, i.e. a previous
super user is no longer a super user.

I filled Release Note with Jerry's description.

When you click on 'Edit' button, you should find 'Release Note' field under
'Hadoop Flags' field.

Thanks

Visibility Labels: issue with storing super users in labels table
-

Attachments: 12644-0.98.patch, HBASE-12644-master-v2.patch,
HBASE-12644-master-v3.patch, HBASE-12644-master.patch

Additionally, there is no mechanism to remove any super user from the labels
table.

We probably should not persist super users in the labels table.
They are in hbase-site.xml and can just stay in labelsCache and used from
labelsCache after retrieval by Visibility Controller.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-9431) Set 'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims


 [ 
https://issues.apache.org/jira/browse/HBASE-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-9431:
-
   Resolution: Fixed
Fix Version/s: 2.0.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks for fixing my mangled posting [~enis] Pushed to branch-1.

 Set  'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims
 ---

 Key: HBASE-9431
 URL: https://issues.apache.org/jira/browse/HBASE-9431
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
 Fix For: 1.0.0, 2.0.0

 Attachments: 9431.txt, hbase-12072_v1.patch, hbase-9431.patch


 HBASE-8450 claimes  'hbase.bulkload.retries.number' is set to 10 when its 
 still 0 ([~jeffreyz] noticed).  Fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk

2014-12-16 Thread Enis Soztutar (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248724#comment-14248724
 ] 

Enis Soztutar commented on HBASE-10201:
---

I don't think we should have this in 1.0.0. I am planning on cutting the RC 
tomorrow, and this seems to be a huge change for the last minute. Can we target 
1.1 instead? 

 Port 'Make flush decisions per column family' to trunk
 --

 Key: HBASE-10201
 URL: https://issues.apache.org/jira/browse/HBASE-10201
 Project: HBase
  Issue Type: Improvement
  Components: wal
Reporter: Ted Yu
Assignee: zhangduo
 Fix For: 1.0.0, 2.0.0

 Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, 
 HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, 
 HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, 
 HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, 
 HBASE-10201_13.patch, HBASE-10201_14.patch, HBASE-10201_15.patch, 
 HBASE-10201_16.patch, HBASE-10201_17.patch, HBASE-10201_18.patch, 
 HBASE-10201_19.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, 
 HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, 
 HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch, 
 compactions.png, count.png, io.png, memstore.png


 Currently the flush decision is made using the aggregate size of all column 
 families. When large and small column families co-exist, this causes many 
 small flushes of the smaller CF. We need to make per-CF flush decisions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Jesse Yates (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jesse Yates updated HBASE-5162:
---
Attachment: hbase-5162-trunk-v12-committed.patch

Attaching patch for actual code committed to trunk.

Thanks for the copious reviews [~apurtell] and hanging in there.

Basic client pushback mechanism
---

Key: HBASE-5162
URL: https://issues.apache.org/jira/browse/HBASE-5162
Project: HBase
Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
Fix For: 1.0.0, 2.0.0

Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch,
hbase-5162-trunk-v10.patch, hbase-5162-trunk-v11.patch,
hbase-5162-trunk-v12-committed.patch, hbase-5162-trunk-v2.patch,
hbase-5162-trunk-v3.patch, hbase-5162-trunk-v4.patch,
hbase-5162-trunk-v5.patch, hbase-5162-trunk-v6.patch,
hbase-5162-trunk-v7.patch, hbase-5162-trunk-v8.patch, java_HBASE-5162.patch

The current blocking we do when we are close to some limits (memstores over
the multiplier factor, too many store files, global memstore memory) is bad,
too coarse and confusing. After hitting HBASE-5161, it really becomes obvious
that we need something better.
I did a little brainstorm with Stack, we came up quickly with two solutions:
- Send some exception to the client, like OverloadedException, that's thrown
when some situation happens like getting past the low memory barrier. It
would be thrown when the client gets a handler and does some check while
putting or deleting. The client would treat this a retryable exception but
ideally wouldn't check .META. for a new location. It could be fancy and have
multiple levels of pushback, like send the exception to 25% of the clients,
and then go up if the situation persists. Should be easy to implement but
we'll be using a lot more IO to send the payload over and over again (but at
least it wouldn't sit in the RS's memory).
- Send a message alongside a successful put or delete to tell the client to
slow down a little, this way we don't have to do back and forth with the
payload between the client and the server. It's a cleaner (I think) but more
involved solution.
In every case the RS should do very obvious things to notify the operators of
this situation, through logs, web UI, metrics, etc.
Other ideas?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk

2014-12-16 Thread Jeffrey Zhong (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248725#comment-14248725
 ] 

Jeffrey Zhong commented on HBASE-10201:
---

Looks good to me(+1) for master branch.  Branch-1 should rely on [~enis]'s 
feedbacks.

 Port 'Make flush decisions per column family' to trunk
 --

 Key: HBASE-10201
 URL: https://issues.apache.org/jira/browse/HBASE-10201
 Project: HBase
  Issue Type: Improvement
  Components: wal
Reporter: Ted Yu
Assignee: zhangduo
 Fix For: 1.0.0, 2.0.0

 Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, 
 HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, 
 HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, 
 HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, 
 HBASE-10201_13.patch, HBASE-10201_14.patch, HBASE-10201_15.patch, 
 HBASE-10201_16.patch, HBASE-10201_17.patch, HBASE-10201_18.patch, 
 HBASE-10201_19.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, 
 HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, 
 HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch, 
 compactions.png, count.png, io.png, memstore.png


 Currently the flush decision is made using the aggregate size of all column 
 families. When large and small column families co-exist, this causes many 
 small flushes of the smaller CF. We need to make per-CF flush decisions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir


[ 
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248729#comment-14248729
 ] 

Varun Saxena commented on HBASE-12645:
--

[~stack], if the flag is false, {{getDefaultRootDirPath()}} calls below 
function. As you can see, if {{dataTestDirOnTestFS}} is null, directory path is 
created/fetched again.
If not, previous value is retrieved. When the flag is true,  we create a new 
path irrespective of whether dataTestDirOnTestFS is null or not.
{code}
public Path getDataTestDirOnTestFS() throws IOException {
if (dataTestDirOnTestFS == null) {
  setupDataTestDirOnTestFS();
}

return dataTestDirOnTestFS;
  }
{code}

 HBaseTestingUtility is using ${$HOME} for rootDir
 -

 Key: HBASE-12645
 URL: https://issues.apache.org/jira/browse/HBASE-12645
 Project: HBase
  Issue Type: Test
  Components: test
Affects Versions: 1.0.0
Reporter: Nick Dimiduk
Assignee: Varun Saxena
Priority: Critical
 Fix For: 1.0.0, 2.0.0

 Attachments: HBASE-12645.002.patch, HBASE-12645.003.patch, 
 HBASE-12645.004.patch, HBASE-12645.004.patch, HBASE-12645.005.patch, 
 HBASE-12645.006.patch, HBASE-12645.006.patch, HBASE-12645.patch


 I noticed this while running tests on branch-1
 {noformat}
 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec  
 FAILURE! - in 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits  Time 
 elapsed: 0.009 sec   ERROR!
 java.io.FileNotFoundException: Destination exists and is not a directory: 
 /homes/hortonnd/hbase
 at 
 org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
 at 
 org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
 at 
 org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
 at 
 org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
 {noformat}
 Either the testing utility has a regression or there's a config regression in 
 this test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=true

2014-12-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248739#comment-14248739
 ] 

Hudson commented on HBASE-12699:


SUCCESS: Integrated in HBase-TRUNK #5929 (See 
[https://builds.apache.org/job/HBase-TRUNK/5929/])
HBASE-12699 undefined method 'setAsyncLogFlush' exception thrown when setting 
DEFERRED_LOG_FLUSH=true (Stephen Jiang) (tedyu: rev 
1359e87b1757dd32d81fa2039af55156de1b3298)
* hbase-shell/src/main/ruby/hbase/admin.rb


 undefined method `setAsyncLogFlush' exception thrown when setting 
 DEFERRED_LOG_FLUSH=true
 --

 Key: HBASE-12699
 URL: https://issues.apache.org/jira/browse/HBASE-12699
 Project: HBase
  Issue Type: Bug
  Components: shell
Affects Versions: 2.0.0, 0.99.2
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
 Fix For: 1.0.0, 2.0.0

 Attachments: HBASE-12699.v1.branch-1.patch, 
 HBASE-12699.v1.master.patch

   Original Estimate: 24h
  Time Spent: 4h
  Remaining Estimate: 1h

 In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, 
 an undefined method `setAsyncLogFlush' exception was thrown.  
 This is due to that DEFERRED_LOG_FLUSH was deprecated and the 
 setAsyncLogFlush method was removed.  It was replaced by DURABILITY.
 DEFERRED_LOG_FLUSH=true is the same as DURABILITY='ASYNC_WAL'
 The default is DURABILITY='SYNC_WAL', which is the same as the default 
 DEFERRED_LOG_FLUSH=false
 We should ask user to use the DURABILITY setting.  In the meantime, for 
 backward compatibility, the hbase shell should still allow setting 
 DEFERRED_LOG_FLUSH.  Internally, instead of calling setAsyncLogFlush, it 
 should call setDurability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248738#comment-14248738
]

Hadoop QA commented on HBASE-5162:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12687543/hbase-5162-trunk-v12-committed.patch
against master branch at commit a411227b0ebf78b4ee8ae7179e162b54734e77de.
ATTACHMENT ID: 12687543

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 19 new
or modified tests.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12100//console

This message is automatically generated.

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=true

[
https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248736#comment-14248736
]

Hadoop QA commented on HBASE-12699:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12687519/HBASE-12699.v1.branch-1.patch
against master branch at commit 96c6b9815ddbc9f2589655df4ad2381af04ac9f8.
ATTACHMENT ID: 12687519

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+# DEFERRED_LOG_FLUSH is deprecated and was replaced by DURABILITY.
To keep backward compatible, it still exists.
+# However, it has to set before DURABILITY so that DURABILITY could
overwrite if both args are set
+# DEFERRED_LOG_FLUSH is deprecated and was replaced by DURABILITY. To
keep backward compatible, it still exists.
+# However, it has to set before DURABILITY so that DURABILITY could
overwrite if both args are set

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//artifact/patchprocess/checkstyle-aggregate.html

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12097//console

This message is automatically generated.

undefined method `setAsyncLogFlush' exception thrown when setting
DEFERRED_LOG_FLUSH=true
--

Key: HBASE-12699
URL: https://issues.apache.org/jira/browse/HBASE-12699
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 2.0.0, 0.99.2
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 1.0.0, 2.0.0

Attachments: HBASE-12699.v1.branch-1.patch,
HBASE-12699.v1.master.patch

Original Estimate: 24h
Time Spent: 4h
Remaining Estimate: 1h

In hbase shell, when trying to set DEFERRED_LOG_FLUSH during

[jira] [Updated] (HBASE-10605) Manage the call timeout in the server


 [ 
https://issues.apache.org/jira/browse/HBASE-10605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-10605:
--
Component/s: rpc

 Manage the call timeout in the server
 -

 Key: HBASE-10605
 URL: https://issues.apache.org/jira/browse/HBASE-10605
 Project: HBase
  Issue Type: Improvement
  Components: regionserver, rpc
Affects Versions: 0.99.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 1.0.0


 Since HBASE-10566, we have an explicit call timeout available in the client.
 We could forward it to the server, and use this information for:
 - if the call is still in the queue, just cancel it
 - if the call is under execution, makes this information available in 
 RpcCallContext (actually change the RpcCallContext#disconnectSince to 
 something more generic), so it can be used by the query under execution to 
 stop its execution
 - in the future, interrupt it to manage the case 'stuck on a dead datanode' 
 or something similar
 - if the operation has finished, don't send the reply to the client, as by 
 definition the client is not interested anymore.
 From this, it will be easy to manage the cancellation: 
 disconnect/timeout/cancellation are similar from a service execution PoV



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Jesse Yates (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Yates updated HBASE-5162:
---
Status: Open  (was: Patch Available)

 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 1.0.0, 2.0.0

 Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, 
 hbase-5162-trunk-v10.patch, hbase-5162-trunk-v11.patch, 
 hbase-5162-trunk-v12-committed.patch, hbase-5162-trunk-v2.patch, 
 hbase-5162-trunk-v3.patch, hbase-5162-trunk-v4.patch, 
 hbase-5162-trunk-v5.patch, hbase-5162-trunk-v6.patch, 
 hbase-5162-trunk-v7.patch, hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12644) Visibility Labels: issue with storing super users in labels table

2014-12-16 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248757#comment-14248757
 ] 

Jerry He commented on HBASE-12644:
--

Got it.  Thanks, Ted.

 Visibility Labels: issue with storing super users in labels table
 -

 Key: HBASE-12644
 URL: https://issues.apache.org/jira/browse/HBASE-12644
 Project: HBase
  Issue Type: Bug
  Components: security
Affects Versions: 0.98.8, 0.99.2
Reporter: Jerry He
Assignee: Jerry He
 Fix For: 1.0.0, 2.0.0

 Attachments: 12644-0.98.patch, HBASE-12644-master-v2.patch, 
 HBASE-12644-master-v3.patch, HBASE-12644-master.patch


 Super users have all the permissions for ACL and Visibility labels.
 They are defined in hbase-site.xml.
 Currently in VisibilityController, we persist super user with their system 
 permission in hbase:labels.
 This makes change in super user difficult.
 There are two issues:
 In the current DefaultVisibilityLabelServiceImpl.addSystemLabel, we only add 
 super user when we initially create the 'system' label.
 No additional update after that even if super user changed. See code for 
 details.
  
 Additionally, there is no mechanism to remove any super user from the labels 
 table.
  
 We probably should not persist super users in the labels table.
 They are in hbase-site.xml and can just stay in labelsCache and used from 
 labelsCache after retrieval by Visibility Controller.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Jesse Yates (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jesse Yates updated HBASE-5162:
---
Attachment: hbase-5162-branch-1-v0.patch

Attaching patch for branch-1, [~enis] - thoughts on getting this in?

Also, [~apurtell], would you like a patch for 0.98 as well? I think it should
be pretty close to 1.0/2.0 version.

Basic client pushback mechanism
---

Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch,
hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch,
hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch,
hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch,
hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch,
hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch,
hbase-5162-trunk-v8.patch, java_HBASE-5162.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Andrew Purtell (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248780#comment-14248780
]

Andrew Purtell commented on HBASE-5162:
---

I have a feeling [~enis] won't take this at this late time for 1.0. What if we
open a branch (soon) for 1.1.

I think we can look at this for 0.98 if we don't break non-private interfaces
and we can get it back into 0.98 from master via any 1.x.

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk


[ 
https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248793#comment-14248793
 ] 

stack commented on HBASE-10201:
---

bq. Can we target 1.1 instead?

Sure.

 Port 'Make flush decisions per column family' to trunk
 --

 Key: HBASE-10201
 URL: https://issues.apache.org/jira/browse/HBASE-10201
 Project: HBase
  Issue Type: Improvement
  Components: wal
Reporter: Ted Yu
Assignee: zhangduo
 Fix For: 1.0.0, 2.0.0

 Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, 
 HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, 
 HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, 
 HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, 
 HBASE-10201_13.patch, HBASE-10201_14.patch, HBASE-10201_15.patch, 
 HBASE-10201_16.patch, HBASE-10201_17.patch, HBASE-10201_18.patch, 
 HBASE-10201_19.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, 
 HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, 
 HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch, 
 compactions.png, count.png, io.png, memstore.png


 Currently the flush decision is made using the aggregate size of all column 
 families. When large and small column families co-exist, this causes many 
 small flushes of the smaller CF. We need to make per-CF flush decisions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-10201) Port 'Make flush decisions per column family' to trunk


 [ 
https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-10201:
--
   Resolution: Fixed
Fix Version/s: (was: 1.0.0)
 Release Note: Adds new flushing policy mechanism. Default, 
org.apache.hadoop.hbase.regionserver.FlushLargeStoresPolicy, will try to avoid 
flushing out the small column families in a region, those whose memstores are  
hbase.hregion.percolumnfamilyflush.size.lower.bound. To restore the old 
behavior of flushes writing out all column families, set 
hbase.regionserver.flush.policy to 
org.apache.hadoop.hbase.regionserver.FlushAllStoresPolicy either in 
hbase-default.xml or on a per-table basis by setting the policy to use with 
HTableDescriptor.getFlushPolicyClassName().
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Pushed to master branch

 Port 'Make flush decisions per column family' to trunk
 --

 Key: HBASE-10201
 URL: https://issues.apache.org/jira/browse/HBASE-10201
 Project: HBase
  Issue Type: Improvement
  Components: wal
Reporter: Ted Yu
Assignee: zhangduo
 Fix For: 2.0.0

 Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, 
 HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, 
 HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, 
 HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, 
 HBASE-10201_13.patch, HBASE-10201_14.patch, HBASE-10201_15.patch, 
 HBASE-10201_16.patch, HBASE-10201_17.patch, HBASE-10201_18.patch, 
 HBASE-10201_19.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, 
 HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, 
 HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch, 
 compactions.png, count.png, io.png, memstore.png


 Currently the flush decision is made using the aggregate size of all column 
 families. When large and small column families co-exist, this causes many 
 small flushes of the smaller CF. We need to make per-CF flush decisions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-12700) Backport 'Make flush decisions per column family' from master

stack created HBASE-12700:
-

 Summary: Backport 'Make flush decisions per column family' from 
master
 Key: HBASE-12700
 URL: https://issues.apache.org/jira/browse/HBASE-12700
 Project: HBase
  Issue Type: Task
Reporter: stack
 Fix For: 1.1.0


Task to backport this nice feature to 1.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk


[ 
https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248816#comment-14248816
 ] 

stack commented on HBASE-10201:
---

I forgot to say thank you [~Apache9] for your persistence on getting this in.

 Port 'Make flush decisions per column family' to trunk
 --

 Key: HBASE-10201
 URL: https://issues.apache.org/jira/browse/HBASE-10201
 Project: HBase
  Issue Type: Improvement
  Components: wal
Reporter: Ted Yu
Assignee: zhangduo
 Fix For: 2.0.0

 Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, 
 HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, 
 HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, 
 HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, 
 HBASE-10201_13.patch, HBASE-10201_14.patch, HBASE-10201_15.patch, 
 HBASE-10201_16.patch, HBASE-10201_17.patch, HBASE-10201_18.patch, 
 HBASE-10201_19.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, 
 HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, 
 HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch, 
 compactions.png, count.png, io.png, memstore.png


 Currently the flush decision is made using the aggregate size of all column 
 families. When large and small column families co-exist, this causes many 
 small flushes of the smaller CF. We need to make per-CF flush decisions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir