[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Abhishek Singh Chouhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Singh Chouhan updated HBASE-17937:
---
Attachment: HBASE-17937.branch-1.002.patch

Addressed comments. Added comments to test class.

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.branch-1.002.patch, HBASE-17937.master.001.patch, 
> HBASE-17937.master.002.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.003.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17667) Implement async flush/compact region methods

2017-04-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974151#comment-15974151
 ] 

Hadoop QA commented on HBASE-17667:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 51s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 5s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 22s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
34s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 13s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
|   | hadoop.hbase.snapshot.TestExportSnapshot |
|   | hadoop.hbase.snapshot.TestSecureExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12863940/HBASE-17667.v2.patch |
| JIRA Issue | HBASE-17667 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 90d6c0fafc48 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 66b616d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6497/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/Pre

[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Abhishek Singh Chouhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Singh Chouhan updated HBASE-17937:
---
Attachment: HBASE-17937.master.003.patch

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.002.patch, HBASE-17937.master.003.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17940) HMaster can not start due to Jasper related classes conflict

2017-04-18 Thread Duo Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17940:
--
Affects Version/s: 2.0.0
 Priority: Blocker  (was: Major)
Fix Version/s: 2.0.0
  Component/s: pom
   dependencies

Make it blocker for 2.0.0.

> HMaster can not start due to Jasper related classes conflict
> 
>
> Key: HBASE-17940
> URL: https://issues.apache.org/jira/browse/HBASE-17940
> Project: HBase
>  Issue Type: Bug
>  Components: dependencies, pom
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Priority: Blocker
> Fix For: 2.0.0
>
>
> I got this
> {noformat}
> java.lang.RuntimeException: Failed construction of Master: class 
> org.apache.hadoop.hbase.master.HMaster.
> at 
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2692)
> at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:235)
> at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at 
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2703)
> Caused by: java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
> at 
> org.apache.jasper.compiler.JspRuntimeContext.(JspRuntimeContext.java:194)
> at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:159)
> at 
> org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:640)
> at 
> org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:419)
> at 
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
> at 
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
> at 
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
> at 
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
> at 
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
> at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
> at 
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> at org.eclipse.jetty.server.Server.start(Server.java:405)
> at 
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
> at 
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart(Server.java:372)
> at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:969)
> at org.apache.hadoop.hbase.http.InfoServer.start(InfoServer.java:100)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1887)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:620)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:461)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2685)
> ... 5 more
> {noformat}
> The problem is that, we have same classes in two jars, 
> javax.servlet.jsp-2.3.2.jar and jasper-compiler-5.5.23.jar, such as 
> org.apache.jasper.Constants and org.apache.jasper.compiler.JspRuntimeContext. 
> Here seems for JspRuntimeContext we use the version from javax.servlet.jsp, 
> but for Constants we use the version from jasper-compiler.
> javax.servlet.jsp is a transitive dependency from j

[jira] [Created] (HBASE-17940) HMaster can not start due to Jasper related classes conflict

2017-04-18 Thread Duo Zhang (JIRA)
Duo Zhang created HBASE-17940:
-

 Summary: HMaster can not start due to Jasper related classes 
conflict
 Key: HBASE-17940
 URL: https://issues.apache.org/jira/browse/HBASE-17940
 Project: HBase
  Issue Type: Bug
Reporter: Duo Zhang


I got this
{noformat}
java.lang.RuntimeException: Failed construction of Master: class 
org.apache.hadoop.hbase.master.HMaster.
at 
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2692)
at 
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:235)
at 
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at 
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2703)
Caused by: java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
at 
org.apache.jasper.compiler.JspRuntimeContext.(JspRuntimeContext.java:194)
at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:159)
at 
org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:640)
at 
org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:419)
at 
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
at 
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
at 
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
at 
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at 
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at 
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:969)
at org.apache.hadoop.hbase.http.InfoServer.start(InfoServer.java:100)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1887)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:620)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:461)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2685)
... 5 more
{noformat}

The problem is that, we have same classes in two jars, 
javax.servlet.jsp-2.3.2.jar and jasper-compiler-5.5.23.jar, such as 
org.apache.jasper.Constants and org.apache.jasper.compiler.JspRuntimeContext. 
Here seems for JspRuntimeContext we use the version from javax.servlet.jsp, but 
for Constants we use the version from jasper-compiler.

javax.servlet.jsp is a transitive dependency from jetty jsp, and 
jasper-compiler is a dependency directly in our pom for hbase-server.

We need to find a way to deal with it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default

2017-04-18 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974093#comment-15974093
 ] 

stack commented on HBASE-15131:
---

No dependency true. Was just expressing what I thought more important but you 
can think multiwal more important... no harm [~carp84]


> Turn on multi-WAL by default
> 
>
> Key: HBASE-15131
> URL: https://issues.apache.org/jira/browse/HBASE-15131
> Project: HBase
>  Issue Type: New Feature
>  Components: wal
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
>
> Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL 
> by default now that it has seen some production use. 
> Most of the known issues has been fixed I believe for replication, metrics 
> etc. See HBASE-14457. 
> Using bounded provider with default 4 (assuming 12 disks). Should we also 
> look at the number of disks from datanode dirs and auto-configure? 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager

2017-04-18 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974088#comment-15974088
 ] 

stack commented on HBASE-14614:
---

Just an update. I've been working this patch on cluster. Interesting bugs. 
Keeping account up in 
https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
 See BUGs section. Two bad ones. 1. when subprocedures are done, they fail 
because their parent finished early (miscounting -- two worker threads can run 
same procedure...), and 2. after crash, for some reason, we do not reschedule 
an odd proc.. trying to figure it. There are others and have been working on 
log cleanup so has narrative; i.e. always log procId along side event change so 
can associate procedure and log message... etc. Will be back w/ more status. 
See doc in meantime if you would like to follow along.

> Procedure v2: Core Assignment Manager
> -
>
> Key: HBASE-14614
> URL: https://issues.apache.org/jira/browse/HBASE-14614
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Stephen Yuan Jiang
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-14614.master.003.patch, 
> HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, 
> HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, 
> HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, 
> HBASE-14614.master.010.patch, HBASE-14614.master.011.patch, 
> HBASE-14614.master.012.patch, HBASE-14614.master.012.patch, 
> HBASE-14614.master.013.patch, HBASE-14614.master.014.patch, 
> HBASE-14614.master.015.patch, HBASE-14614.master.017.patch, 
> HBASE-14614.master.018.patch, HBASE-14614.master.019.patch, 
> HBASE-14614.master.020.patch, HBASE-14614.master.021.patch, 
> HBASE-14614.master.022.patch, HBASE-14614.master.023.patch, 
> HBASE-14614.master.024.patch, HBASE-14614.master.025.patch, 
> HBASE-14614.master.026.patch, HBASE-14614.master.027.patch, 
> HBASE-14614.master.028.patch, HBASE-14614.master.029.patch, 
> HBASE-14614.master.030.patch, HBASE-14614.master.031.patch, 
> HBASE-14614.master.032.patch, HBASE-14614.master.033.patch, 
> HBASE-14614.master.034.patch, HBASE-14614.master.035.patch, 
> HBASE-14614.master.036.patch, HBASE-14614.master.037.patch, 
> HBASE-14614.master.038.patch
>
>
> New AssignmentManager implemented using proc-v2.
>  - AssignProcedure handle assignment operation
>  - UnassignProcedure handle unassign operation
>  - MoveRegionProcedure handle move/balance operation
> Concurrent Assign operations are batched together and sent to the balancer
> Concurrent Assign and Unassign operation ready to be sent to the RS are 
> batched together
> This patch is an intermediate state where we add the new AM as 
> AssignmentManager2() to the master, to be reached by tests. but the new AM 
> will not be integrated with the rest of the system. Only new am unit-tests 
> will exercise the new assigment manager. The integration with the master code 
> is part of HBASE-14616



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Duo Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974078#comment-15974078
 ] 

Duo Zhang commented on HBASE-17937:
---

[~lhofhansl] As [~abhishek.chouhan] said, in master we sync wal before 
modifying memstore so...

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Comment Edited] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Abhishek Singh Chouhan (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974057#comment-15974057
 ] 

Abhishek Singh Chouhan edited comment on HBASE-17937 at 4/19/17 4:41 AM:
-

[~lhofhansl] In master the order of operations in mvcc is different from 
branch-1. We do not apply to memstore first and then if wal sync fails, 
rollback. Rather we sync to wal first and then write to memstore, hence 
updating the size just after that step.
[~enis] I'll make the changes and put up a new patch. Initial idea of the test 
was to have a coprocessor thats slow and at the same time flush should happen, 
since that was turning a bit difficult thought to trigger flush from the 
coprocessor itself :D


was (Author: abhishek.chouhan):
[~lhofhansl] In master the order of operations in mvcc is different from 
branch-1. We do not apply to memstore first and then if wal sync fails, 
rollback. Rather we sync to wal first and then write to memstore, hence 
updating the size just after that step.
[~enis] I'll make the changes and put up a new patch. Initial idea of the test 
was to have a coprocessor thats slow and at the same time flush should happen, 
since that was turning a bit difficult thought to trigger flush from the 
coprocess itself :D

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Abhishek Singh Chouhan (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974057#comment-15974057
 ] 

Abhishek Singh Chouhan commented on HBASE-17937:


[~lhofhansl] In master the order of operations in mvcc is different from 
branch-1. We do not apply to memstore first and then if wal sync fails, 
rollback. Rather we sync to wal first and then write to memstore, hence 
updating the size just after that step.
[~enis] I'll make the changes and put up a new patch. Initial idea of the test 
was to have a coprocessor thats slow and at the same time flush should happen, 
since that was turning a bit difficult thought to trigger flush from the 
coprocess itself :D

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it

2017-04-18 Thread ramkrishna.s.vasudevan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-16438:
---
Attachment: HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch

I think after commits only two tests had failures and both had been fixed. LEt 
me try with QA once before I recommit this.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it

2017-04-18 Thread ramkrishna.s.vasudevan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-16438:
---
Status: Patch Available  (was: Open)

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it

2017-04-18 Thread ramkrishna.s.vasudevan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-16438:
---
Status: Open  (was: Patch Available)

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

2017-04-18 Thread ramkrishna.s.vasudevan (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974040#comment-15974040
 ] 

ramkrishna.s.vasudevan commented on HBASE-16438:


>From the small tests 
I got these failures
{code}
Tests in error:
  TestBackupHFileCleaner.setUpBeforeClass:66 » IO Shutting down
org.apache.hadoop.hbase.master.balancer.TestRegionLocationFinder.org.apache.hadoop.hbase.master.balancer.TestRegionLocationFinder
  Run 1: TestRegionLocationFinder.setUpBeforeClass:58 » IO Shutting down
  Run 2: TestRegionLocationFinder.tearDownAfterClass:77 NullPointer

  TestLockManager.setupCluster:77 » IO Shutting down
  TestLockProcedure.setupCluster:103 » IO Shutting down
  TestProcedureManager.setupBeforeClass:54 » IO Shutting down
  TestHRegionFileSystem.testBlockStoragePolicy:77 » IO Shutting down
{code}
which seems to be unrelated.
>From the medium tests I got
{code}
Failed tests:
  TestRegionLoad.testRegionLoad:103->compareRegionLoads:110 No of regionLoads 
from clusterStatus and regionloads from RS doesn't match expected:<8> but 
was:<12>
{code}
>From large tests I got no failures.

The TestReversibleScanners and TestFSHLog that were failing have been fixed in 
the latest patch. It was nothing but it needed ChunkCreator to be initialized.
bq.Looks like the above assignment can be moved after currentInstance is 
created.
Nope. We need a new type of Chunkcreator so that it is able to retire the 
chunks immediately. So we nullify the current Instance and recreate the 
chunkCreator instance and again assign the original one back to the INSTANCE 
variable. If needed I can rename from 'currentInstance' to 'oldInstance'.


> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default

2017-04-18 Thread Yu Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974038#comment-15974038
 ] 

Yu Li commented on HBASE-15131:
---

I think they are two different dimensions: singleWAL v.s. multiWAL, and syncWAL 
v.s. asyncWAL. We have been living with syncWAL and multiWAL online for over 
two years and it proves to be stable, also some performance number (both 
benchmark and online) in HBASE-14457.

And yes, would be better with asyncWAL, but no dependency between these two I 
guess? (smile)

> Turn on multi-WAL by default
> 
>
> Key: HBASE-15131
> URL: https://issues.apache.org/jira/browse/HBASE-15131
> Project: HBase
>  Issue Type: New Feature
>  Components: wal
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
>
> Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL 
> by default now that it has seen some production use. 
> Most of the known issues has been fixed I believe for replication, metrics 
> etc. See HBASE-14457. 
> Using bounded provider with default 4 (assuming 12 disks). Should we also 
> look at the number of disks from datanode dirs and auto-configure? 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974036#comment-15974036
 ] 

Hudson commented on HBASE-17925:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2884 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2884/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
719a30b11a7fddc5f8e5318064d8fb9ef5eab7cb)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-16215) clean up references for EOM release lines

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974035#comment-15974035
 ] 

Hudson commented on HBASE-16215:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2884 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2884/])
HBASE-16215 clean up of ref guide and site for EOM versions. (busbey: rev 
a8e6f33791d787eaf9c1bfe63aa7f3266f25268b)
* (edit) src/main/asciidoc/_chapters/configuration.adoc
* (edit) src/main/asciidoc/_chapters/upgrading.adoc
* (edit) src/main/asciidoc/_chapters/developer.adoc
* (edit) src/main/asciidoc/_chapters/cp.adoc
* (edit) src/main/site/site.xml
* (edit) src/main/asciidoc/_chapters/community.adoc


> clean up references for EOM release lines
> -
>
> Key: HBASE-16215
> URL: https://issues.apache.org/jira/browse/HBASE-16215
> Project: HBase
>  Issue Type: Task
>  Components: community, website
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Attachments: HBASE-16215.0.patch, HBASE-16215.1.patch
>
>
> I've seen us state a few places that 1.0.z is EOM. We should clean up 
> remaining references
> * remove 1.0.z artifact from dist.apache
> * remove it from the ref guide (java prereqs, hadoop prereqs, RM list)
> * remove unreleased 1.0.z versions from JIRA
> * archive released 1.0.z versions in JIRA
> edit: expanded to include
> * 0.94.y [first EOM 
> DISCUSS|https://lists.apache.org/thread.html/547058fc59cb48130b35c80f86d41c28038195c99f5ed94e834e291c@%3Cdev.hbase.apache.org%3E],
>  [second EOM 
> DISCUSS|https://lists.apache.org/thread.html/5ed4ae42ee9be61f963dd0a49f6896fb7771cd8d319d2d86b3c3cc76@%3Cdev.hbase.apache.org%3E]
> * 0.98.y [EOM 
> DISCUSS|https://lists.apache.org/thread.html/fa968458e0f7f224cf906a259448a207b1230548ce6f2c0d50ceeb12@%3Cdev.hbase.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Resolved] (HBASE-17929) Add more options for PE tool

2017-04-18 Thread Duo Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-17929.
---
Resolution: Fixed

Pushed the addendum to master.

> Add more options for PE tool
> 
>
> Key: HBASE-17929
> URL: https://issues.apache.org/jira/browse/HBASE-17929
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance, test
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17929-addendum.patch, HBASE-17929.patch
>
>
> For better evaluating scan performance.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default

2017-04-18 Thread Duo Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974029#comment-15974029
 ] 

Duo Zhang commented on HBASE-15131:
---

Yeah this could be done together with the async wal test.

> Turn on multi-WAL by default
> 
>
> Key: HBASE-15131
> URL: https://issues.apache.org/jira/browse/HBASE-15131
> Project: HBase
>  Issue Type: New Feature
>  Components: wal
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
>
> Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL 
> by default now that it has seen some production use. 
> Most of the known issues has been fixed I believe for replication, metrics 
> etc. See HBASE-14457. 
> Using bounded provider with default 4 (assuming 12 disks). Should we also 
> look at the number of disks from datanode dirs and auto-configure? 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17929) Add more options for PE tool

2017-04-18 Thread Duo Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17929:
--
Attachment: HBASE-17929-addendum.patch

> Add more options for PE tool
> 
>
> Key: HBASE-17929
> URL: https://issues.apache.org/jira/browse/HBASE-17929
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance, test
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17929-addendum.patch, HBASE-17929.patch
>
>
> For better evaluating scan performance.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Reopened] (HBASE-17929) Add more options for PE tool

2017-04-18 Thread Duo Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang reopened HBASE-17929:
---

Forget to add scan read type support.

> Add more options for PE tool
> 
>
> Key: HBASE-17929
> URL: https://issues.apache.org/jira/browse/HBASE-17929
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance, test
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17929.patch
>
>
> For better evaluating scan performance.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default

2017-04-18 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974017#comment-15974017
 ] 

stack commented on HBASE-15131:
---

Oh, you want to see if this will help you...  Maybe [~busbey] has his more 
recent results?

> Turn on multi-WAL by default
> 
>
> Key: HBASE-15131
> URL: https://issues.apache.org/jira/browse/HBASE-15131
> Project: HBase
>  Issue Type: New Feature
>  Components: wal
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
>
> Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL 
> by default now that it has seen some production use. 
> Most of the known issues has been fixed I believe for replication, metrics 
> etc. See HBASE-14457. 
> Using bounded provider with default 4 (assuming 12 disks). Should we also 
> look at the number of disks from datanode dirs and auto-configure? 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default

2017-04-18 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974016#comment-15974016
 ] 

stack commented on HBASE-15131:
---

Not sure [~carp84] I'd rather work on the async dfs client-based WAL 
implementaiton try and figure why that isn't going fast First.

> Turn on multi-WAL by default
> 
>
> Key: HBASE-15131
> URL: https://issues.apache.org/jira/browse/HBASE-15131
> Project: HBase
>  Issue Type: New Feature
>  Components: wal
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
>
> Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL 
> by default now that it has seen some production use. 
> Most of the known issues has been fixed I believe for replication, metrics 
> etc. See HBASE-14457. 
> Using bounded provider with default 4 (assuming 12 disks). Should we also 
> look at the number of disks from datanode dirs and auto-configure? 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17675) ReplicationEndpoint should choose new sinks if a SaslException occurs

2017-04-18 Thread Nick Dimiduk (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-17675:
-
Summary: ReplicationEndpoint should choose new sinks if a SaslException 
occurs  (was: ReplicationEndpoint should choose new sinks if a SaslException 
occurs )

> ReplicationEndpoint should choose new sinks if a SaslException occurs
> -
>
> Key: HBASE-17675
> URL: https://issues.apache.org/jira/browse/HBASE-17675
> Project: HBase
>  Issue Type: Bug
>Reporter: churro morales
>Assignee: churro morales
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.10
>
> Attachments: HBASE-17675.patch
>
>
> We had an issue where a regionserver on our destination side failed to 
> refresh the keytabs.  The source side's replication got stuck because the 
> HBaseInterClusterReplicationEndpoint only chooses new sinks if there happens 
> to be a ConnectException but the SaslException is an IOException, which does 
> not choose new sinks.  
> I'll put up a patch to check this exception and choose new sinks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17667) Implement async flush/compact region methods

2017-04-18 Thread Zheng Hu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zheng Hu updated HBASE-17667:
-
Attachment: HBASE-17667.v2.patch

Patch v2:
1.  split CompletableFuture compactRegionServer(ServerName sn, boolean 
major) into two method :
CompletableFuture compactRegionServer(ServerName sn) ;
CompletableFuture majorCompactRegionServer(ServerName sn) ;

2. add a UT for both compactRegionServer and majorCompactRegionServer. 

> Implement  async  flush/compact region methods
> --
>
> Key: HBASE-17667
> URL: https://issues.apache.org/jira/browse/HBASE-17667
> Project: HBase
>  Issue Type: Sub-task
>  Components: Admin, asyncclient, Client
>Affects Versions: 2.0.0
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-17667.v1.patch, HBASE-17667.v1.patch, 
> HBASE-17667.v2.patch
>
>
> Implement following methods for async admin: 
> {code}
> 1. flush table ; 
> 2. flush region; 
> 3. compact table;
> 4. compact region;
> 5. compact region server; 
> 6. major compact for table; 
> 7. major compact for region; 
> 8. major compact for CF;
> 9. major compact for specific region and specific CF; 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17513) Thrift Server 1 uses different QOP settings than RPC and Thrift Server 2 and can easily be misconfigured so there is no encryption when the operator expects it.

2017-04-18 Thread Nick Dimiduk (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-17513:
-
Fix Version/s: (was: 1.1.10)
   1.1.11

> Thrift Server 1 uses different QOP settings than RPC and Thrift Server 2 and 
> can easily be misconfigured so there is no encryption when the operator 
> expects it.
> 
>
> Key: HBASE-17513
> URL: https://issues.apache.org/jira/browse/HBASE-17513
> Project: HBase
>  Issue Type: Bug
>  Components: documentation, security, Thrift, Usability
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0, 1.2.6, 1.3.2, 1.1.11
>
>
> As of HBASE-14400 the setting {{hbase.thrift.security.qop}} was unified to 
> behave the same as the general HBase RPC protection. However, this only 
> happened for the Thrift2 server. The Thrift server found in the thrift 
> package (aka Thrift Server 1) still hard codes the old configs of 'auth', 
> 'auth-int', and 'auth-conf'.
> Additionally, these Quality of Protection (qop) settings are used only by the 
> SASL transport. If a user configures the HBase Thrift Server to make use of 
> the HTTP transport (to enable doAs proxying e.g. for Hue) then a QOP setting 
> of 'privacy' or 'auth-conf' won't get them encryption as expected.
> We should
> 1) update {{hbase-thrift/src/main/.../thrift/ThriftServerRunner}} to rely on 
> {{SaslUtil}} to use the same 'authentication', 'integrity', 'privacy' configs 
> in a backward compatible way
> 2) also have ThriftServerRunner warn when both {{hbase.thrift.security.qop}} 
> and {{hbase.regionserver.thrift.http}} are set, since the latter will cause 
> the former to be ignored. (users should be directed to 
> {{hbase.thrift.ssl.enabled}} and related configs to ensure their transport is 
> encrypted when using the HTTP transport.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17885) Backport HBASE-15871 to branch-1

2017-04-18 Thread Nick Dimiduk (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-17885:
-
Fix Version/s: (was: 1.1.9)
   1.1.11

> Backport HBASE-15871 to branch-1
> 
>
> Key: HBASE-17885
> URL: https://issues.apache.org/jira/browse/HBASE-17885
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 1.3.1, 1.2.5, 1.1.8
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.2.6, 1.3.2, 1.1.11
>
>
> Will try to rebase the branch-1 patch at the earliest. Hope the fix versions 
> are correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default

2017-04-18 Thread Yu Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974006#comment-15974006
 ] 

Yu Li commented on HBASE-15131:
---

Should we revive this thread for 2.0.0?

[~stack] [~Apache9] could you also take a look here? Thanks.

> Turn on multi-WAL by default
> 
>
> Key: HBASE-15131
> URL: https://issues.apache.org/jira/browse/HBASE-15131
> Project: HBase
>  Issue Type: New Feature
>  Components: wal
>Reporter: Enis Soztutar
> Fix For: 2.0.0
>
>
> Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL 
> by default now that it has seen some production use. 
> Most of the known issues has been fixed I believe for replication, metrics 
> etc. See HBASE-14457. 
> Using bounded provider with default 4 (assuming 12 disks). Should we also 
> look at the number of disks from datanode dirs and auto-configure? 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server

2017-04-18 Thread Nick Dimiduk (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk updated HBASE-17534:
-
Fix Version/s: (was: 1.1.10)
   1.1.11

> SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
> 
>
> Key: HBASE-17534
> URL: https://issues.apache.org/jira/browse/HBASE-17534
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11
>
> Attachments: HBASE-17534.001.branch-1.patch, 
> HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch
>
>
> While writing some tests against 1.x, I noticed that what should have been a 
> DoNotRetryIOException sent to the client from a RegionServer was getting 
> retried until it reached the hbase client retries limit.
> Upon inspection, I found that the SecureBulkLoadClient was wrapping all 
> Exceptions from the RPC as an IOException. I believe this is creating a case 
> where the RPC system doesn't notice that there's a DNRIOException wrapped 
> beneath it, thinking it's a transient error.
> This results in clients having to wait for the retry limit to be reached 
> before they get acknowledgement that something failed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973981#comment-15973981
 ] 

Hudson commented on HBASE-17925:


SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #123 (See 
[https://builds.apache.org/job/HBase-1.2-JDK7/123/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
d0d162d1f7594ef0454d0aed60c391f268e68dbd)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973971#comment-15973971
 ] 

Hudson commented on HBASE-17925:


SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #119 (See 
[https://builds.apache.org/job/HBase-1.2-JDK8/119/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
d0d162d1f7594ef0454d0aed60c391f268e68dbd)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973965#comment-15973965
 ] 

Hudson commented on HBASE-17925:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #146 (See 
[https://builds.apache.org/job/HBase-1.3-JDK7/146/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
acbba3749ad02c62ae4b45e7ba07321ae9400d8a)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17915) Implement async replication admin methods

2017-04-18 Thread Guanghao Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973963#comment-15973963
 ] 

Guanghao Zhang commented on HBASE-17915:


Timeout ut not related. And I try these ut on my computer and passed locally. 
Will commit it if no objection.

> Implement async replication admin methods
> -
>
> Key: HBASE-17915
> URL: https://issues.apache.org/jira/browse/HBASE-17915
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17915.master.001.patch, 
> HBASE-17915.master.001.patch, HBASE-17915.master.001.patch, 
> HBASE-17915.master.002.patch, HBASE-17915.master.003.patch, 
> HBASE-17915.master.004.patch, HBASE-17915.master.005.patch, 
> HBASE-17915.master.005.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973915#comment-15973915
 ] 

Hudson commented on HBASE-17925:


FAILURE: Integrated in Jenkins build HBase-1.4 #697 (See 
[https://builds.apache.org/job/HBase-1.4/697/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
6ea6915245ed276f441a8686daf2c9a66cf9115f)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17939) Backport HBASE-17914 to branch-1

2017-04-18 Thread Duo Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17939:
--
Component/s: regionserver
 Compaction

> Backport HBASE-17914 to branch-1
> 
>
> Key: HBASE-17939
> URL: https://issues.apache.org/jira/browse/HBASE-17939
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Reporter: Duo Zhang
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (HBASE-17939) Backport HBASE-17914 to branch-1

2017-04-18 Thread Duo Zhang (JIRA)
Duo Zhang created HBASE-17939:
-

 Summary: Backport HBASE-17914 to branch-1
 Key: HBASE-17939
 URL: https://issues.apache.org/jira/browse/HBASE-17939
 Project: HBase
  Issue Type: Sub-task
Reporter: Duo Zhang






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973909#comment-15973909
 ] 

Hudson commented on HBASE-17925:


SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1857 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1857/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
c50d40a15d66959590b9ccd50667a190b3a94059)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction

2017-04-18 Thread Duo Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17914:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to master.

Thanks [~stack] and [~ram_krish] for reviewing.

> Create a new reader instead of cloning a new StoreFile when compaction
> --
>
> Key: HBASE-17914
> URL: https://issues.apache.org/jira/browse/HBASE-17914
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17914.patch, HBASE-17914-v1.patch, 
> HBASE-17914-v1.patch, HBASE-17914-v2.patch, HBASE-17914-v3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-15296) Break out writer and reader from StoreFile

2017-04-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973883#comment-15973883
 ] 

Hadoop QA commented on HBASE-15296:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 12s {color} 
| {color:red} HBASE-15296 does not apply to branch-1. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12806778/HBASE-15296-branch-1-v1.patch
 |
| JIRA Issue | HBASE-15296 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6496/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Break out writer and reader from StoreFile
> --
>
> Key: HBASE-15296
> URL: https://issues.apache.org/jira/browse/HBASE-15296
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-15296-branch-1.1.patch, 
> HBASE-15296-branch-1.2.patch, HBASE-15296-branch-1.patch, 
> HBASE-15296-branch-1.patch, HBASE-15296-branch-1-v1.patch, 
> HBASE-15296-master.patch, HBASE-15296-master-v2.patch, 
> HBASE-15296-master-v3.patch, HBASE-15296-master-v4.patch, 
> HBASE-15296-master-v5.patch
>
>
> StoreFile.java is trending to become a monolithic class, it's ~1800 lines. 
> Would it make sense to break out reader and writer (~500 lines each) into 
> separate files.
> We are doing so many different things in a single class: comparators, reader, 
> writer, other stuff; and it hurts readability a lot, to the point that just 
> reading through a piece of code require scrolling up and down to see which 
> level (reader/writer/base class level) it belongs to. These small-small 
> things really don't help while trying to understanding the code. There are 
> good reasons we don't do these often (affects existing patches, needs to be 
> done for all branches, etc). But this and a few other classes can really use 
> a single iteration of refactoring to make things a lot better.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-16314) Retry on table snapshot failure

2017-04-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973880#comment-15973880
 ] 

Hadoop QA commented on HBASE-16314:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m 25s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
41m 58s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 136m 59s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
44s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 207m 56s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.util.TestHBaseFsckTwoRS |
|   | hadoop.hbase.util.TestHBaseFsckReplicas |
| Timed out junit tests | 
org.apache.hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
|   | org.apache.hadoop.hbase.regionserver.TestCompactSplitThread |
|   | org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss |
|   | org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
|   | org.apache.hadoop.hbase.regionserver.wal.TestLogRolling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12863902/HBASE-16314-v5.patch |
| JIRA Issue | HBASE-16314 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux acdb02381f09 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / a8e6f33 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6495/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCo

[jira] [Commented] (HBASE-15296) Break out writer and reader from StoreFile

2017-04-18 Thread Duo Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973877#comment-15973877
 ] 

Duo Zhang commented on HBASE-15296:
---

Any updates on backporting to branch-1 here? The backporting of HBASE-17914 
will be much easier if we could have this patch in branch-1.

Thanks.

> Break out writer and reader from StoreFile
> --
>
> Key: HBASE-15296
> URL: https://issues.apache.org/jira/browse/HBASE-15296
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-15296-branch-1.1.patch, 
> HBASE-15296-branch-1.2.patch, HBASE-15296-branch-1.patch, 
> HBASE-15296-branch-1.patch, HBASE-15296-branch-1-v1.patch, 
> HBASE-15296-master.patch, HBASE-15296-master-v2.patch, 
> HBASE-15296-master-v3.patch, HBASE-15296-master-v4.patch, 
> HBASE-15296-master-v5.patch
>
>
> StoreFile.java is trending to become a monolithic class, it's ~1800 lines. 
> Would it make sense to break out reader and writer (~500 lines each) into 
> separate files.
> We are doing so many different things in a single class: comparators, reader, 
> writer, other stuff; and it hurts readability a lot, to the point that just 
> reading through a piece of code require scrolling up and down to see which 
> level (reader/writer/base class level) it belongs to. These small-small 
> things really don't help while trying to understanding the code. There are 
> good reasons we don't do these often (affects existing patches, needs to be 
> done for all branches, etc). But this and a few other classes can really use 
> a single iteration of refactoring to make things a lot better.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction

2017-04-18 Thread Duo Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973872#comment-15973872
 ] 

Duo Zhang commented on HBASE-17914:
---

Backporting the patch to branch-1 is not straight-forward as StoreFileReader is 
still called StoreFile.reader. Will open a backporting issue to do it.

> Create a new reader instead of cloning a new StoreFile when compaction
> --
>
> Key: HBASE-17914
> URL: https://issues.apache.org/jira/browse/HBASE-17914
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17914.patch, HBASE-17914-v1.patch, 
> HBASE-17914-v1.patch, HBASE-17914-v2.patch, HBASE-17914-v3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973865#comment-15973865
 ] 

Hudson commented on HBASE-17925:


FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #158 (See 
[https://builds.apache.org/job/HBase-1.3-JDK8/158/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
acbba3749ad02c62ae4b45e7ba07321ae9400d8a)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17924) Consider sorting the row order when processing multi() ops before taking rowlocks

2017-04-18 Thread Allan Yang (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973863#comment-15973863
 ] 

Allan Yang commented on HBASE-17924:


{quote}
Test failure looks related to patch.
org.apache.hadoop.hbase.wal.WALSplitter$MutationReplay defines 
compareTo(WALSplitter$MutationReplay) and uses Object.equals() At 
WALSplitter.java:Object.equals() At WALSplitter.java:[line 2301]
{quote}
Yes, There is a bug in the patch, I'm working on it

> Consider sorting the row order when processing multi() ops before taking 
> rowlocks
> -
>
> Key: HBASE-17924
> URL: https://issues.apache.org/jira/browse/HBASE-17924
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0, 1.1.8
>Reporter: Andrew Purtell
>Assignee: Allan Yang
> Fix For: 2.0.0
>
> Attachments: HBASE-17924.patch, HBASE-17924.v0.patch
>
>
> When processing a batch mutation, we take row locks in whatever order the 
> mutations were added to the multi op by the client.
>  
> {noformat}
> RSRpcServices#multi -> RSRpcServices#mutateRows -> HRegion#mutateRow -> 
> HRegion#mutateRowsWithLocks -> HRegion#processRowsWithLocks
> {noformat}
> Or
> {noformat}
> RSRpcServices#multi -> RSRpcServices#doNonAtomicRegionMutation ->
>   HRegion#get 
> | HRegion#append 
> | HRegion#increment 
> | HRegionServer#doBatchOp -> HRegion#batchMutate -> 
> HRegion#doMiniBatchMutation
> {noformat}
>  
> multi() is fed by client APIs that accept a RowMutations object containing 
> actions for multiple rows. The container for ops inside RowMutations is an 
> ArrayList, which doesn't change the ordering of objects added to it. The 
> protobuf implementation of the messages for multi ops do not reorder the list 
> of actions. When processing multi ops we iterate over the actions in the 
> order rehydrated from protobuf.
> We should discuss sorting the order of ops by row key when processing multi() 
> ops before taking row locks. Does this make lock ordering more predictable 
> for server side operations? Yes, but potentially surprising for the client, 
> right? Is there any legitimate reason we should take locks out of row key 
> sorted order because the client has structured the request as such?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973860#comment-15973860
 ] 

Hudson commented on HBASE-17925:


FAILURE: Integrated in Jenkins build HBase-1.1-JDK8 #1941 (See 
[https://builds.apache.org/job/HBase-1.1-JDK8/1941/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
c50d40a15d66959590b9ccd50667a190b3a94059)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17867) Implement async procedure RPC API(list/exec/abort/isFinished)

2017-04-18 Thread Zheng Hu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973847#comment-15973847
 ] 

Zheng Hu commented on HBASE-17867:
--

Nice explanation for async isProcedureFinished method.   UT failed because of  
forgetting add category annotation .  Will fix it and upload a new patch.


> Implement async procedure RPC API(list/exec/abort/isFinished)
> -
>
> Key: HBASE-17867
> URL: https://issues.apache.org/jira/browse/HBASE-17867
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-17867.v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973828#comment-15973828
 ] 

Hudson commented on HBASE-17925:


SUCCESS: Integrated in Jenkins build HBase-1.2-IT #858 (See 
[https://builds.apache.org/job/HBase-1.2-IT/858/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
d0d162d1f7594ef0454d0aed60c391f268e68dbd)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17867) Implement async procedure RPC API(list/exec/abort/isFinished)

2017-04-18 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973813#comment-15973813
 ] 

stack commented on HBASE-17867:
---

Skimmed patch. Looking good. Is failure related?

> Implement async procedure RPC API(list/exec/abort/isFinished)
> -
>
> Key: HBASE-17867
> URL: https://issues.apache.org/jira/browse/HBASE-17867
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-17867.v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17936) Refine sum endpoint example in ref guide

2017-04-18 Thread Xiang Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973798#comment-15973798
 ] 

Xiang Li commented on HBASE-17936:
--

Thanks  [~stack] !

> Refine sum endpoint example in ref guide
> 
>
> Key: HBASE-17936
> URL: https://issues.apache.org/jira/browse/HBASE-17936
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17936.master.000.patch
>
>
> Refine chapter 90.2, the example java code of sum endpoint of 
> http://hbase.apache.org/book.html#cp_example
> - error
> - typo
> - genericity
> - indent



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16465) Disable region splits and merges, balancer during full backup

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-16465:
--
Fix Version/s: (was: HBASE-7912)
   2.0.0

> Disable region splits and merges, balancer during full backup
> -
>
> Key: HBASE-16465
> URL: https://issues.apache.org/jira/browse/HBASE-16465
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16465-v1.patch, HBASE-16465-v2.patch, 
> HBASE-16465-v3.patch, HBASE-16465-v4.patch, HBASE-16465-v5.patch, 
> HBASE-16465-v6.patch
>
>
> Incorporate HBASE-15128
> Balancer, catalog janitor and region normalizer should be disabled as well 
> during full backup



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973767#comment-15973767
 ] 

Hudson commented on HBASE-17925:


SUCCESS: Integrated in Jenkins build HBase-1.3-IT #27 (See 
[https://builds.apache.org/job/HBase-1.3-IT/27/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev 
acbba3749ad02c62ae4b45e7ba07321ae9400d8a)
* (edit) pom.xml


> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Jonathan Hsieh (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973754#comment-15973754
 ] 

Jonathan Hsieh commented on HBASE-17925:


Thanks for the review Ted.  Committed.

> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2

2017-04-18 Thread Jonathan Hsieh (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-17925:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.3.2
   1.2.6
   1.1.9
   1.4.0
   2.0.0
   Status: Resolved  (was: Patch Available)

> mvn assembly:single fails against hadoop3-alpha2
> 
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
>  Issue Type: Sub-task
>  Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use 
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single 
> -DskipTests
> {code}
> This currently fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-15227) HBase Backup Phase 3: Fault tolerance (client/server) support

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-15227:
--
Description: 
System must be tolerant to faults: 

# Backup operations MUST be atomic (no partial completion state in the backup 
system table)
# Process must detect any type of failures which can result in a data loss 
(partial backup or partial restore) 
# Proper system table state restore and cleanup must be done in case of a 
failure
# Additional utility to repair backup system table and corresponding file 
system cleanup must be implemented

  was:
System must be tolerant to faults: 

# Backup operations MUST be atomic (no partial completion state in the backup 
system table)
# Process must detect any type of failures which can result in a data loss 
(partial backup) in a backup destination
# Proper system table state restore and cleanup must be done in case of a 
failure
# Additional utility to repair backup system table and corresponding file 
system cleanup must be implemented


> HBase Backup Phase 3: Fault tolerance (client/server) support
> -
>
> Key: HBASE-15227
> URL: https://issues.apache.org/jira/browse/HBASE-15227
> Project: HBase
>  Issue Type: Task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-15227-v3.patch, HBASE-15277-v1.patch
>
>
> System must be tolerant to faults: 
> # Backup operations MUST be atomic (no partial completion state in the backup 
> system table)
> # Process must detect any type of failures which can result in a data loss 
> (partial backup or partial restore) 
> # Proper system table state restore and cleanup must be done in case of a 
> failure
> # Additional utility to repair backup system table and corresponding file 
> system cleanup must be implemented



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-15227) HBase Backup Phase 3: Fault tolerance (client/server) support

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-15227:
--
Description: 
System must be tolerant to faults: 

# Backup operations MUST be atomic (no partial completion state in the backup 
system table)
# Process must detect any type of failures which can result in a data loss 
(partial backup) in a backup destination
# Proper system table state restore and cleanup must be done in case of a 
failure
# Additional utility to repair backup system table and corresponding file 
system cleanup must be implemented

  was:
System must be tolerant to faults: 

# Backup operations MUST be atomic (no partial completion state in system table)
# Process must detect any type of failures which can result in a data loss 
(partial backup) in a backup destination
# Proper system table state restore and cleanup must be done in case of a 
failure
# Additional utility to repair backup system table and corresponding file 
system cleanup must be implemented


> HBase Backup Phase 3: Fault tolerance (client/server) support
> -
>
> Key: HBASE-15227
> URL: https://issues.apache.org/jira/browse/HBASE-15227
> Project: HBase
>  Issue Type: Task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-15227-v3.patch, HBASE-15277-v1.patch
>
>
> System must be tolerant to faults: 
> # Backup operations MUST be atomic (no partial completion state in the backup 
> system table)
> # Process must detect any type of failures which can result in a data loss 
> (partial backup) in a backup destination
> # Proper system table state restore and cleanup must be done in case of a 
> failure
> # Additional utility to repair backup system table and corresponding file 
> system cleanup must be implemented



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17924) Consider sorting the row order when processing multi() ops before taking rowlocks

2017-04-18 Thread Enis Soztutar (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973693#comment-15973693
 ] 

Enis Soztutar commented on HBASE-17924:
---

Good point, we do this sorting in MultiRowMutationEndpoint: 
{code}
  // set of rows to lock, sorted to avoid deadlocks
  SortedSet rowsToLock = new TreeSet<>(Bytes.BYTES_COMPARATOR);
  List mutateRequestList = request.getMutationRequestList();
  List mutations = new ArrayList<>(mutateRequestList.size());
  for (MutationProto m : mutateRequestList) {
mutations.add(ProtobufUtil.toMutation(m));
  }
{code}
Makes sense to do it in the regular batch path as well. 

> Consider sorting the row order when processing multi() ops before taking 
> rowlocks
> -
>
> Key: HBASE-17924
> URL: https://issues.apache.org/jira/browse/HBASE-17924
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0, 1.1.8
>Reporter: Andrew Purtell
>Assignee: Allan Yang
> Fix For: 2.0.0
>
> Attachments: HBASE-17924.patch, HBASE-17924.v0.patch
>
>
> When processing a batch mutation, we take row locks in whatever order the 
> mutations were added to the multi op by the client.
>  
> {noformat}
> RSRpcServices#multi -> RSRpcServices#mutateRows -> HRegion#mutateRow -> 
> HRegion#mutateRowsWithLocks -> HRegion#processRowsWithLocks
> {noformat}
> Or
> {noformat}
> RSRpcServices#multi -> RSRpcServices#doNonAtomicRegionMutation ->
>   HRegion#get 
> | HRegion#append 
> | HRegion#increment 
> | HRegionServer#doBatchOp -> HRegion#batchMutate -> 
> HRegion#doMiniBatchMutation
> {noformat}
>  
> multi() is fed by client APIs that accept a RowMutations object containing 
> actions for multiple rows. The container for ops inside RowMutations is an 
> ArrayList, which doesn't change the ordering of objects added to it. The 
> protobuf implementation of the messages for multi ops do not reorder the list 
> of actions. When processing multi ops we iterate over the actions in the 
> order rehydrated from protobuf.
> We should discuss sorting the order of ops by row key when processing multi() 
> ops before taking row locks. Does this make lock ordering more predictable 
> for server side operations? Yes, but potentially surprising for the client, 
> right? Is there any legitimate reason we should take locks out of row key 
> sorted order because the client has structured the request as such?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973689#comment-15973689
 ] 

Hadoop QA commented on HBASE-17937:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 31s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
31s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s 
{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
38s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 7s 
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s 
{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
30m 8s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 165m 9s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
22s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 218m 58s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 
|
|   | hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
|   | hadoop.hbase.regionserver.TestCompactionInDeadRegionServer |
|   | hadoop.hbase.snapshot.TestExportSnapshot |
|   | hadoop.hbase.util.TestHBaseFsck |
|   | hadoop.hbase.client.TestClientOperationInterrupt |
|   | hadoop.hbase.master.TestMasterBalanceThrottling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:e01ee2f |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12863870/HBASE-17937.branch-1.001.patch
 |
| JIRA Issue | HBASE-17937 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux ed2ee2b8f7ad 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016 
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 / c8c2e07 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6491/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6491/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/6491/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test 

[jira] [Commented] (HBASE-17867) Implement async procedure RPC API(list/exec/abort/isFinished)

2017-04-18 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973688#comment-15973688
 ] 

stack commented on HBASE-17867:
---

Why not an async isProcedureFinished? It involves a trip to the Master, Master 
does internal query, perhaps a GC hits at this exact time Better if we 
don't tie up the client while all this work is going on?

> Implement async procedure RPC API(list/exec/abort/isFinished)
> -
>
> Key: HBASE-17867
> URL: https://issues.apache.org/jira/browse/HBASE-17867
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-17867.v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-16181) Allow snapshot of hbase:backup table

2017-04-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973681#comment-15973681
 ] 

Hadoop QA commented on HBASE-16181:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
67m 34s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 12s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 50s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
40s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 105m 13s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12863889/HBASE-16181-v3.patch |
| JIRA Issue | HBASE-16181 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 8f034b45a94a 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016 
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / d15f75b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6493/testReport/ |
| modules | C: hbase-common hbase-client U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6493/console |
| Powered by | Apache Yetu

[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Enis Soztutar (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973674#comment-15973674
 ] 

Enis Soztutar commented on HBASE-17937:
---

Makes sense. 

For the patch, you can remove commented-out code: 
{code}
// TEST_UTIL.getConfiguration().setLong(HConstants.HREGION_MEMSTORE_FLUSH_SIZE, 
100);
{code}

Also rename the {{SlowRegionObserver}}, since it is not a slow coprocessor, but 
instead triggers flush itself. 

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction

2017-04-18 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973667#comment-15973667
 ] 

stack commented on HBASE-17914:
---

+1

Lets try it.


> Create a new reader instead of cloning a new StoreFile when compaction
> --
>
> Key: HBASE-17914
> URL: https://issues.apache.org/jira/browse/HBASE-17914
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction, regionserver
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17914.patch, HBASE-17914-v1.patch, 
> HBASE-17914-v1.patch, HBASE-17914-v2.patch, HBASE-17914-v3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-16181) Allow snapshot of hbase:backup table

2017-04-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973635#comment-15973635
 ] 

Hadoop QA commented on HBASE-16181:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 6s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 13s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
13s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 34s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12863896/HBASE-16181-v4.patch |
| JIRA Issue | HBASE-16181 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 5aad303b2ec9 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / d15f75b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6494/testReport/ |
| modules | C: hbase-common hbase-client U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6494/console |
| Powered by | Apach

[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973624#comment-15973624
 ] 

Hadoop QA commented on HBASE-17937:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 12s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 112m 40s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 45s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
|   | hadoop.hbase.snapshot.TestSecureExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12863873/HBASE-17937.master.002.patch
 |
| JIRA Issue | HBASE-17937 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 793e72be90b3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / d15f75b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6492/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/6492/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6492/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6492/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Memstore size becomes negative in case of expensive postPut/Del

[jira] [Updated] (HBASE-16314) Retry on table snapshot failure

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-16314:
--
Attachment: HBASE-16314-v5.patch

v5. Rebased to current master.

cc: [~tedyu]

> Retry on table snapshot failure
> ---
>
> Key: HBASE-16314
> URL: https://issues.apache.org/jira/browse/HBASE-16314
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16314-v1.patch, HBASE-16314-v2.patch, 
> HBASE-16314-v3.patch, HBASE-16314-v4.patch, HBASE-16314-v5.patch
>
>
> Table snapshot (during full backup) can fail with NotServingRegionException 
> (splits, AM region moves). We need to add retry logic here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator

2017-04-18 Thread Steve Loughran (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973578#comment-15973578
 ] 

Steve Loughran commented on HBASE-17877:


bq Also while looking at some UnsignedBytes function for the guava version we 
were using I noticed that guava v14.0.1 also uses this implementation so 
probably hadoop borrowed it from there ..

I was about to say "no it doesn't", but then I saw the comment at the top, 
"This is borrowed and slightly modified from Guava's". Been in there a long 
time though (since 2012), so it's history is lost. x86 pert is what really 
matters, though we don't want to be pathologically antisocial to the other CPU 
arches, not just for the sake of PPC, but for when Arm goes mainstream in the 
DC.



> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, 
> 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, 
> HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, 
> HBASE-17877.master.001.patch, HBASE-17877.master.002.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is 
> faster - dramatically faster in some cases.
> Patch forthcoming.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16181) Allow snapshot of hbase:backup table

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-16181:
--
Attachment: HBASE-16181-v4.patch

v4.

Do we do any security checks during snapshots? If, yes, then we are fine - only 
super user will be able to run snapshot on  a system table.

> Allow snapshot of hbase:backup table
> 
>
> Key: HBASE-16181
> URL: https://issues.apache.org/jira/browse/HBASE-16181
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch, 
> HBASE-16181-v3.patch, HBASE-16181-v4.patch
>
>
> Snapshot of HBase system tables is not supported, we need either move 
> hbase:backup into different name space or fix snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-16181) Allow snapshot of hbase:backup table

2017-04-18 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973545#comment-15973545
 ] 

Ted Yu commented on HBASE-16181:


{code}
536   public void setSnapshottable(boolean value) {
{code}
Returning TableName (this) would allow chaining.

Is there any security concern with this new API ?

> Allow snapshot of hbase:backup table
> 
>
> Key: HBASE-16181
> URL: https://issues.apache.org/jira/browse/HBASE-16181
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch, 
> HBASE-16181-v3.patch
>
>
> Snapshot of HBase system tables is not supported, we need either move 
> hbase:backup into different name space or fix snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17935) NPE when using PreemptiveFastFailInterceptor with HBaseAdmin client

2017-04-18 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973528#comment-15973528
 ] 

Ted Yu commented on HBASE-17935:


For unit test, refer to TestFastFail where HBASE_CLIENT_FAST_FAIL_MODE_ENABLED 
is set true.

See if namespaceExists() triggers the NPE.

> NPE when using PreemptiveFastFailInterceptor with HBaseAdmin client
> ---
>
> Key: HBASE-17935
> URL: https://issues.apache.org/jira/browse/HBASE-17935
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Robert Yokota
>Assignee: Robert Yokota
>Priority: Minor
> Attachments: HBASE-17935.master.001.patch
>
>
> When setting hbase.client.fast.fail.mode.enabled=true and using the admin 
> client, got a NPE:
> java.lang.NullPointerException
> at 
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at 
> org.apache.hadoop.hbase.client.PreemptiveFastFailInterceptor.intercept(PreemptiveFastFailInterceptor.java:115)
> at 
> org.apache.hadoop.hbase.client.PreemptiveFastFailInterceptor.intercept(PreemptiveFastFailInterceptor.java:376)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:135)
> at 
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at 
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at 
> org.apache.hadoop.hbase.client.HBaseAdmin.getNamespaceDescriptor(HBaseAdmin.java:3009)
> at 
> com.yammer.calmie.store.hbase.HBaseConnectionFactory.namespaceExists(HBaseConnectionFactory.java:146)
> at 
> com.yammer.calmie.store.hbase.HBaseConnectionFactory.createHTable(HBaseConnectionFactory.java:116)
> at 
> com.yammer.calmie.store.hbase.HBaseConnectionFactory.declareTable(HBaseConnectionFactory.java:97)
> at 
> com.yammer.feedie.core.index.calmie.MessageCalmieIndex.(MessageCalmieIndex.java:46)
> at com.yammer.feedie.core.MessageStore.init(MessageStore.java:108)
> at com.yammer.feedie.core.MessageStore.(MessageStore.java:89)
> at com.yammer.feedie.FeedieService.setup(FeedieService.java:148)
> at com.yammer.feedie.FeedieService.run(FeedieService.java:315)
> at com.yammer.feedie.FeedieService.run(FeedieService.java:65)
> at 
> com.yammer.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:38)
> at 
> com.yammer.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:58)
> at com.yammer.dropwizard.cli.Cli.run(Cli.java:53)
> at com.yammer.dropwizard.Service.run(Service.java:61)
> at com.yammer.feedie.FeedieService.main(FeedieService.java:84)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16314) Retry on table snapshot failure

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-16314:
--
Labels: backup  (was: )

> Retry on table snapshot failure
> ---
>
> Key: HBASE-16314
> URL: https://issues.apache.org/jira/browse/HBASE-16314
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16314-v1.patch, HBASE-16314-v2.patch, 
> HBASE-16314-v3.patch, HBASE-16314-v4.patch
>
>
> Table snapshot (during full backup) can fail with NotServingRegionException 
> (splits, AM region moves). We need to add retry logic here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16314) Retry on table snapshot failure

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-16314:
--
Fix Version/s: (was: HBASE-7912)
   2.0.0

> Retry on table snapshot failure
> ---
>
> Key: HBASE-16314
> URL: https://issues.apache.org/jira/browse/HBASE-16314
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16314-v1.patch, HBASE-16314-v2.patch, 
> HBASE-16314-v3.patch, HBASE-16314-v4.patch
>
>
> Table snapshot (during full backup) can fail with NotServingRegionException 
> (splits, AM region moves). We need to add retry logic here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-16181) Allow snapshot of hbase:backup table

2017-04-18 Thread Vladimir Rodionov (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973521#comment-15973521
 ] 

Vladimir Rodionov commented on HBASE-16181:
---

cc: [~tedyu], [~chenheng]

> Allow snapshot of hbase:backup table
> 
>
> Key: HBASE-16181
> URL: https://issues.apache.org/jira/browse/HBASE-16181
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch, 
> HBASE-16181-v3.patch
>
>
> Snapshot of HBase system tables is not supported, we need either move 
> hbase:backup into different name space or fix snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16181) Allow snapshot of hbase:backup table

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-16181:
--
Status: Patch Available  (was: Reopened)

> Allow snapshot of hbase:backup table
> 
>
> Key: HBASE-16181
> URL: https://issues.apache.org/jira/browse/HBASE-16181
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch, 
> HBASE-16181-v3.patch
>
>
> Snapshot of HBase system tables is not supported, we need either move 
> hbase:backup into different name space or fix snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16181) Allow snapshot of hbase:backup table

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-16181:
--
Attachment: HBASE-16181-v3.patch

v3. Introduces TableName.isSnapshottable/ TableName.setSnapshottable

By default, system tables are not snapshottable, but it can be overridden with 
the new API. 

No more checks on system tables during restore from snapshots.

As since, new functionality is not exposed to command line tools, there is no 
way to create snapshots for system tables using command line tools.

> Allow snapshot of hbase:backup table
> 
>
> Key: HBASE-16181
> URL: https://issues.apache.org/jira/browse/HBASE-16181
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch, 
> HBASE-16181-v3.patch
>
>
> Snapshot of HBase system tables is not supported, we need either move 
> hbase:backup into different name space or fix snapshots.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973451#comment-15973451
 ] 

Hadoop QA commented on HBASE-17937:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
57m 39s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 55s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
14s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 13s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12863866/HBASE-17937.master.002.patch
 |
| JIRA Issue | HBASE-17937 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux f6009d4a7afa 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016 
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / d15f75b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6490/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/6490/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6490/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6490/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---

[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Lars Hofhansl (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973407#comment-15973407
 ] 

Lars Hofhansl commented on HBASE-17937:
---

+1 on branch-1 patch. (nice to get rid of the call in the finally block!)
For the master patch, shouldn't we also move the size update after the sync of 
the wal? [~Apache9], [~abhishek.chouhan]

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973398#comment-15973398
 ] 

Ted Yu commented on HBASE-17937:


lgtm

nit:
{code}
+  threwIOE = true;
{code}
Rename variable IOEThrown

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread churro morales (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973367#comment-15973367
 ] 

churro morales commented on HBASE-17937:


correct, and if you can't flush the memstore then you could potentially get 
into a state where you are waiting trying to allocate space in the MSlab which 
is done under row locks. 

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Abhishek Singh Chouhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Singh Chouhan updated HBASE-17937:
---
Attachment: HBASE-17937.master.002.patch

Added category annotation to the test.

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Abhishek Singh Chouhan (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973339#comment-15973339
 ] 

Abhishek Singh Chouhan commented on HBASE-17937:


In the patch for branch-1, have added the 
this.addAndGetGlobalMemstoreSize(addedSize) call just after step 7 (wal sync) 
and removed the update from finally block. After step 7 we are not going to 
rollback the memstore (we set doRollbackMemstore=false), hence we can update 
the memstore size there, before step 7 we don't want to update the memstore 
size anyways since we'll do a rollback of the memstore in the finally block. 
[~Apache9] [~tedyu] [~apurtell] [~lhofhansl]


> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Abhishek Singh Chouhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Singh Chouhan updated HBASE-17937:
---
Attachment: HBASE-17937.branch-1.001.patch

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch, 
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (HBASE-17938) General fault - tolerance framework for backup/restore operations

2017-04-18 Thread Vladimir Rodionov (JIRA)
Vladimir Rodionov created HBASE-17938:
-

 Summary: General fault - tolerance framework for backup/restore 
operations
 Key: HBASE-17938
 URL: https://issues.apache.org/jira/browse/HBASE-17938
 Project: HBase
  Issue Type: Sub-task
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov


The framework must take care of all general types of failures during backup/ 
restore and restore system to the original state in case of a failure.

That won't solve all the possible issues  but we have a separate JIRAs for them 
as a sub-tasks of HBASE-15277



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Resolved] (HBASE-15226) HBase Backup/Restore Phase 3: Add FSM to Backup state

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov resolved HBASE-15226.
---
Resolution: Later

> HBase Backup/Restore Phase 3: Add FSM to Backup state
> -
>
> Key: HBASE-15226
> URL: https://issues.apache.org/jira/browse/HBASE-15226
> Project: HBase
>  Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>  Labels: backup
> Fix For: HBASE-7912
>
>
> Finate State Machine. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17924) Consider sorting the row order when processing multi() ops before taking rowlocks

2017-04-18 Thread Andrew Purtell (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973270#comment-15973270
 ] 

Andrew Purtell commented on HBASE-17924:


Test failure looks related to patch.

org.apache.hadoop.hbase.wal.WALSplitter$MutationReplay defines 
compareTo(WALSplitter$MutationReplay) and uses Object.equals() At 
WALSplitter.java:Object.equals() At WALSplitter.java:[line 2301]

> Consider sorting the row order when processing multi() ops before taking 
> rowlocks
> -
>
> Key: HBASE-17924
> URL: https://issues.apache.org/jira/browse/HBASE-17924
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0, 1.1.8
>Reporter: Andrew Purtell
>Assignee: Allan Yang
> Fix For: 2.0.0
>
> Attachments: HBASE-17924.patch, HBASE-17924.v0.patch
>
>
> When processing a batch mutation, we take row locks in whatever order the 
> mutations were added to the multi op by the client.
>  
> {noformat}
> RSRpcServices#multi -> RSRpcServices#mutateRows -> HRegion#mutateRow -> 
> HRegion#mutateRowsWithLocks -> HRegion#processRowsWithLocks
> {noformat}
> Or
> {noformat}
> RSRpcServices#multi -> RSRpcServices#doNonAtomicRegionMutation ->
>   HRegion#get 
> | HRegion#append 
> | HRegion#increment 
> | HRegionServer#doBatchOp -> HRegion#batchMutate -> 
> HRegion#doMiniBatchMutation
> {noformat}
>  
> multi() is fed by client APIs that accept a RowMutations object containing 
> actions for multiple rows. The container for ops inside RowMutations is an 
> ArrayList, which doesn't change the ordering of objects added to it. The 
> protobuf implementation of the messages for multi ops do not reorder the list 
> of actions. When processing multi ops we iterate over the actions in the 
> order rehydrated from protobuf.
> We should discuss sorting the order of ops by row key when processing multi() 
> ops before taking row locks. Does this make lock ordering more predictable 
> for server side operations? Yes, but potentially surprising for the client, 
> right? Is there any legitimate reason we should take locks out of row key 
> sorted order because the client has structured the request as such?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Resolved] (HBASE-17542) Move backup system table into separate namespace

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov resolved HBASE-17542.
---
Resolution: Won't Fix

Currently, the backup system table is in HBase system namespace and let it be 
there.

> Move backup system table into separate namespace
> 
>
> Key: HBASE-17542
> URL: https://issues.apache.org/jira/browse/HBASE-17542
> Project: HBase
>  Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: HBASE-7912
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Resolved] (HBASE-14440) Restore to snapshot

2017-04-18 Thread Vladimir Rodionov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov resolved HBASE-14440.
---
Resolution: Later

> Restore to snapshot
> ---
>
> Key: HBASE-14440
> URL: https://issues.apache.org/jira/browse/HBASE-14440
> Project: HBase
>  Issue Type: New Feature
>Reporter: Vladimir Rodionov
>  Labels: backup
> Fix For: HBASE-7912
>
>
> Good to have feature: restore backup to snapshot. This will allow to massage 
> data with custom M/R job over snapshot. Restore time range only, or 
> particular key range (ranges).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Abhishek Singh Chouhan (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973257#comment-15973257
 ] 

Abhishek Singh Chouhan commented on HBASE-17937:


Have moved the size update to be just after step 5 and removed from finally as 
suggested by [~Apache9] since we don't do the rollback now with the chenged 
mvcc.

[~lhofhansl] Agreed that this is a temporary condition till the time we return 
from the coprocessor call and proceed to the finally part, however if we have 
multiple such requests that seem to take a large amount of time or are stuck, 
like our observation recently we can possibly run into large negatives of 
memstore sizes (lets say -x mb) so the next flush will happen at x+memstore 
flush size which will take more time and so forth. 

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.master.001.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Abhishek Singh Chouhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Singh Chouhan updated HBASE-17937:
---
Attachment: HBASE-17937.master.002.patch

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.master.001.patch, 
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Comment Edited] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Lars Hofhansl (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973189#comment-15973189
 ] 

Lars Hofhansl edited comment on HBASE-17937 at 4/18/17 6:06 PM:


Agreed. I think it's just [~abhishek.chouhan]'s fix, with the first size update 
moved right after the logsync.


was (Author: lhofhansl):
Agreed. I think it's just [~abhishek.chouhan]'s fix, which the first size 
update move right after the logsync.

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.master.001.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Lars Hofhansl (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973189#comment-15973189
 ] 

Lars Hofhansl commented on HBASE-17937:
---

Agreed. I think it's just [~abhishek.chouhan]'s fix, which the first size 
update move right after the logsync.

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.master.001.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Andrew Purtell (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973172#comment-15973172
 ] 

Andrew Purtell commented on HBASE-17937:


Let's fix this. The warning gets your attention and demands a fix, even if it's 
just movement of accounting updates to a better place in the code flow. 

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.master.001.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread Lars Hofhansl (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973163#comment-15973163
 ] 

Lars Hofhansl commented on HBASE-17937:
---

So we looked at the code here. I think this is only a temporary condition like 
this:
1. doMiniBatchMutation starts and writes things into the memstore
2. the background flusher flushes the memstore. Decrementing the memstore size.
3. doMiniBatchMutation finishes. Incrementing the memstore size.

So we only have an issue between #2 and #3. When doMiniBatchMutation finishes 
the size will be correct.
I do _not_ think it's a big problem, just a spurious log warning.

If we did not want that, we should move the incrementing of the memstore size 
right after the sync of the wal in doMiniBatchMutation (inside the MVCC 
transaction).

> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.master.001.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17936) Refine sum endpoint example in ref guide

2017-04-18 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973151#comment-15973151
 ] 

Hudson commented on HBASE-17936:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2882 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2882/])
HBASE-17936 Refine sum endpoint example in ref guide (stack: rev 
d15f75b3cfc5de4def04e94cbb965fd7f578dc34)
* (edit) src/main/asciidoc/_chapters/cp.adoc


> Refine sum endpoint example in ref guide
> 
>
> Key: HBASE-17936
> URL: https://issues.apache.org/jira/browse/HBASE-17936
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17936.master.000.patch
>
>
> Refine chapter 90.2, the example java code of sum endpoint of 
> http://hbase.apache.org/book.html#cp_example
> - error
> - typo
> - genericity
> - indent



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it

2017-04-18 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973144#comment-15973144
 ] 

Ted Yu commented on HBASE-16438:


Mind highlighting how you fixed the tests ?

Thanks

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it

2017-04-18 Thread ramkrishna.s.vasudevan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-16438:
---
Status: Open  (was: Patch Available)

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it

2017-04-18 Thread ramkrishna.s.vasudevan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-16438:
---
Status: Patch Available  (was: Open)

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it

2017-04-18 Thread ramkrishna.s.vasudevan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-16438:
---
Attachment: HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch

Resubmitting for QA. I have run all the test cases in the local linux box. The 
large tests are in progress. By the time will submit for one QA run also.

> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, 
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, 
> HBASE-16438_addendum.patch, HBASE-16438.patch, 
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, 
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call

2017-04-18 Thread churro morales (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973092#comment-15973092
 ] 

churro morales commented on HBASE-17937:


if you get a negative memstore size the problem is you can't flush or close the 
region.  Is there any code path where after a negative memstore size a flush is 
possible?


> Memstore size becomes negative in case of expensive postPut/Delete 
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.master.001.patch
>
>
> We ran into a situation where the memstore size became negative due to 
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update 
> the memstore size in the finally block of doMiniBatchMutate, however a queued 
> flush can be triggered during the coprocessor calls(if they are taking time 
> eg. index updates) since we have released the locks and advanced mvcc at this 
> point. The flush will turn the memstore size negative since the value 
> subtracted is the actual value flushed from stores. The negative value 
> impacts the future flushes amongst others that depend on memstore size.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Resolved] (HBASE-17936) Refine sum endpoint example in ref guide

2017-04-18 Thread stack (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-17936.
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0

Pushed. Thank you for fixup.

> Refine sum endpoint example in ref guide
> 
>
> Key: HBASE-17936
> URL: https://issues.apache.org/jira/browse/HBASE-17936
> Project: HBase
>  Issue Type: Bug
>  Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17936.master.000.patch
>
>
> Refine chapter 90.2, the example java code of sum endpoint of 
> http://hbase.apache.org/book.html#cp_example
> - error
> - typo
> - genericity
> - indent



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator

2017-04-18 Thread stack (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973007#comment-15973007
 ] 

stack commented on HBASE-17877:
---

+1

> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, 
> 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, 
> HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, 
> HBASE-17877.master.001.patch, HBASE-17877.master.002.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is 
> faster - dramatically faster in some cases.
> Patch forthcoming.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (HBASE-15583) Any HTableDescriptor we give out should be immutable

2017-04-18 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972957#comment-15972957
 ] 

Hadoop QA commented on HBASE-15583:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 11s 
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 11s 
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 21 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 52s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m 
45s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 22s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m 
39s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
56m 38s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 19s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 5s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 126m 18s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 14s 
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 23s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 33s 
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 187m 7s {color} 
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 2m 41s 
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:bla

  1   2   >