[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-17937: --- Attachment: HBASE-17937.branch-1.002.patch Addressed comments. Added comments to test class. > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.branch-1.002.patch, HBASE-17937.master.001.patch, > HBASE-17937.master.002.patch, HBASE-17937.master.002.patch, > HBASE-17937.master.003.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17667) Implement async flush/compact region methods
[
https://issues.apache.org/jira/browse/HBASE-17667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974151#comment-15974151
]
Hadoop QA commented on HBASE-17667:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 51s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 22s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | hadoop.hbase.snapshot.TestExportSnapshot |
| | hadoop.hbase.snapshot.TestSecureExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863940/HBASE-17667.v2.patch |
| JIRA Issue | HBASE-17667 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 90d6c0fafc48 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 66b616d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6497/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/Pre
[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-17937: --- Attachment: HBASE-17937.master.003.patch > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, > HBASE-17937.master.002.patch, HBASE-17937.master.003.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17940) HMaster can not start due to Jasper related classes conflict
[
https://issues.apache.org/jira/browse/HBASE-17940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-17940:
--
Affects Version/s: 2.0.0
Priority: Blocker (was: Major)
Fix Version/s: 2.0.0
Component/s: pom
dependencies
Make it blocker for 2.0.0.
> HMaster can not start due to Jasper related classes conflict
>
>
> Key: HBASE-17940
> URL: https://issues.apache.org/jira/browse/HBASE-17940
> Project: HBase
> Issue Type: Bug
> Components: dependencies, pom
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Priority: Blocker
> Fix For: 2.0.0
>
>
> I got this
> {noformat}
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster.
> at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2692)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:235)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2703)
> Caused by: java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
> at
> org.apache.jasper.compiler.JspRuntimeContext.(JspRuntimeContext.java:194)
> at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:159)
> at
> org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:640)
> at
> org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:419)
> at
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
> at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> at org.eclipse.jetty.server.Server.start(Server.java:405)
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> at org.eclipse.jetty.server.Server.doStart(Server.java:372)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:969)
> at org.apache.hadoop.hbase.http.InfoServer.start(InfoServer.java:100)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1887)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:620)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:461)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2685)
> ... 5 more
> {noformat}
> The problem is that, we have same classes in two jars,
> javax.servlet.jsp-2.3.2.jar and jasper-compiler-5.5.23.jar, such as
> org.apache.jasper.Constants and org.apache.jasper.compiler.JspRuntimeContext.
> Here seems for JspRuntimeContext we use the version from javax.servlet.jsp,
> but for Constants we use the version from jasper-compiler.
> javax.servlet.jsp is a transitive dependency from j
[jira] [Created] (HBASE-17940) HMaster can not start due to Jasper related classes conflict
Duo Zhang created HBASE-17940:
-
Summary: HMaster can not start due to Jasper related classes
conflict
Key: HBASE-17940
URL: https://issues.apache.org/jira/browse/HBASE-17940
Project: HBase
Issue Type: Bug
Reporter: Duo Zhang
I got this
{noformat}
java.lang.RuntimeException: Failed construction of Master: class
org.apache.hadoop.hbase.master.HMaster.
at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2692)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:235)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2703)
Caused by: java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
at
org.apache.jasper.compiler.JspRuntimeContext.(JspRuntimeContext.java:194)
at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:159)
at
org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:640)
at
org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:419)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.hadoop.hbase.http.HttpServer.start(HttpServer.java:969)
at org.apache.hadoop.hbase.http.InfoServer.start(InfoServer.java:100)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1887)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:620)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:461)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2685)
... 5 more
{noformat}
The problem is that, we have same classes in two jars,
javax.servlet.jsp-2.3.2.jar and jasper-compiler-5.5.23.jar, such as
org.apache.jasper.Constants and org.apache.jasper.compiler.JspRuntimeContext.
Here seems for JspRuntimeContext we use the version from javax.servlet.jsp, but
for Constants we use the version from jasper-compiler.
javax.servlet.jsp is a transitive dependency from jetty jsp, and
jasper-compiler is a dependency directly in our pom for hbase-server.
We need to find a way to deal with it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default
[ https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974093#comment-15974093 ] stack commented on HBASE-15131: --- No dependency true. Was just expressing what I thought more important but you can think multiwal more important... no harm [~carp84] > Turn on multi-WAL by default > > > Key: HBASE-15131 > URL: https://issues.apache.org/jira/browse/HBASE-15131 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Enis Soztutar > Fix For: 2.0.0 > > > Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL > by default now that it has seen some production use. > Most of the known issues has been fixed I believe for replication, metrics > etc. See HBASE-14457. > Using bounded provider with default 4 (assuming 12 disks). Should we also > look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974088#comment-15974088 ] stack commented on HBASE-14614: --- Just an update. I've been working this patch on cluster. Interesting bugs. Keeping account up in https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit# See BUGs section. Two bad ones. 1. when subprocedures are done, they fail because their parent finished early (miscounting -- two worker threads can run same procedure...), and 2. after crash, for some reason, we do not reschedule an odd proc.. trying to figure it. There are others and have been working on log cleanup so has narrative; i.e. always log procId along side event change so can associate procedure and log message... etc. Will be back w/ more status. See doc in meantime if you would like to follow along. > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-14614.master.003.patch, > HBASE-14614.master.004.patch, HBASE-14614.master.005.patch, > HBASE-14614.master.006.patch, HBASE-14614.master.007.patch, > HBASE-14614.master.008.patch, HBASE-14614.master.009.patch, > HBASE-14614.master.010.patch, HBASE-14614.master.011.patch, > HBASE-14614.master.012.patch, HBASE-14614.master.012.patch, > HBASE-14614.master.013.patch, HBASE-14614.master.014.patch, > HBASE-14614.master.015.patch, HBASE-14614.master.017.patch, > HBASE-14614.master.018.patch, HBASE-14614.master.019.patch, > HBASE-14614.master.020.patch, HBASE-14614.master.021.patch, > HBASE-14614.master.022.patch, HBASE-14614.master.023.patch, > HBASE-14614.master.024.patch, HBASE-14614.master.025.patch, > HBASE-14614.master.026.patch, HBASE-14614.master.027.patch, > HBASE-14614.master.028.patch, HBASE-14614.master.029.patch, > HBASE-14614.master.030.patch, HBASE-14614.master.031.patch, > HBASE-14614.master.032.patch, HBASE-14614.master.033.patch, > HBASE-14614.master.034.patch, HBASE-14614.master.035.patch, > HBASE-14614.master.036.patch, HBASE-14614.master.037.patch, > HBASE-14614.master.038.patch > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974078#comment-15974078 ] Duo Zhang commented on HBASE-17937: --- [~lhofhansl] As [~abhishek.chouhan] said, in master we sync wal before modifying memstore so... > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, > HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974057#comment-15974057 ] Abhishek Singh Chouhan edited comment on HBASE-17937 at 4/19/17 4:41 AM: - [~lhofhansl] In master the order of operations in mvcc is different from branch-1. We do not apply to memstore first and then if wal sync fails, rollback. Rather we sync to wal first and then write to memstore, hence updating the size just after that step. [~enis] I'll make the changes and put up a new patch. Initial idea of the test was to have a coprocessor thats slow and at the same time flush should happen, since that was turning a bit difficult thought to trigger flush from the coprocessor itself :D was (Author: abhishek.chouhan): [~lhofhansl] In master the order of operations in mvcc is different from branch-1. We do not apply to memstore first and then if wal sync fails, rollback. Rather we sync to wal first and then write to memstore, hence updating the size just after that step. [~enis] I'll make the changes and put up a new patch. Initial idea of the test was to have a coprocessor thats slow and at the same time flush should happen, since that was turning a bit difficult thought to trigger flush from the coprocess itself :D > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, > HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974057#comment-15974057 ] Abhishek Singh Chouhan commented on HBASE-17937: [~lhofhansl] In master the order of operations in mvcc is different from branch-1. We do not apply to memstore first and then if wal sync fails, rollback. Rather we sync to wal first and then write to memstore, hence updating the size just after that step. [~enis] I'll make the changes and put up a new patch. Initial idea of the test was to have a coprocessor thats slow and at the same time flush should happen, since that was turning a bit difficult thought to trigger flush from the coprocess itself :D > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, > HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16438: --- Attachment: HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch I think after commits only two tests had failures and both had been fixed. LEt me try with QA once before I recommit this. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_addendum.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16438: --- Status: Patch Available (was: Open) > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_addendum.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16438: --- Status: Open (was: Patch Available) > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_addendum.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974040#comment-15974040
]
ramkrishna.s.vasudevan commented on HBASE-16438:
>From the small tests
I got these failures
{code}
Tests in error:
TestBackupHFileCleaner.setUpBeforeClass:66 » IO Shutting down
org.apache.hadoop.hbase.master.balancer.TestRegionLocationFinder.org.apache.hadoop.hbase.master.balancer.TestRegionLocationFinder
Run 1: TestRegionLocationFinder.setUpBeforeClass:58 » IO Shutting down
Run 2: TestRegionLocationFinder.tearDownAfterClass:77 NullPointer
TestLockManager.setupCluster:77 » IO Shutting down
TestLockProcedure.setupCluster:103 » IO Shutting down
TestProcedureManager.setupBeforeClass:54 » IO Shutting down
TestHRegionFileSystem.testBlockStoragePolicy:77 » IO Shutting down
{code}
which seems to be unrelated.
>From the medium tests I got
{code}
Failed tests:
TestRegionLoad.testRegionLoad:103->compareRegionLoads:110 No of regionLoads
from clusterStatus and regionloads from RS doesn't match expected:<8> but
was:<12>
{code}
>From large tests I got no failures.
The TestReversibleScanners and TestFSHLog that were failing have been fixed in
the latest patch. It was nothing but it needed ChunkCreator to be initialized.
bq.Looks like the above assignment can be moved after currentInstance is
created.
Nope. We need a new type of Chunkcreator so that it is able to retire the
chunks immediately. So we nullify the current Instance and recreate the
chunkCreator instance and again assign the original one back to the INSTANCE
variable. If needed I can rename from 'currentInstance' to 'oldInstance'.
> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments:
> HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch,
> HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch,
> HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch,
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch,
> HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch,
> HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch,
> HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch,
> HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch,
> HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch,
> HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch,
> HBASE-16438_addendum.patch, HBASE-16438.patch,
> MemstoreChunkCell_memstoreChunkCreator_oldversion.patch,
> MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was
> created, the id of the chunk be embedded in it so that when doing flattening
> we can use the chunk id as a meta data. More details will follow once the
> initial tasks are completed.
> Why we need to embed the chunkid in the Cell is described by [~anastas] in
> this remark over in parent issue
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default
[ https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974038#comment-15974038 ] Yu Li commented on HBASE-15131: --- I think they are two different dimensions: singleWAL v.s. multiWAL, and syncWAL v.s. asyncWAL. We have been living with syncWAL and multiWAL online for over two years and it proves to be stable, also some performance number (both benchmark and online) in HBASE-14457. And yes, would be better with asyncWAL, but no dependency between these two I guess? (smile) > Turn on multi-WAL by default > > > Key: HBASE-15131 > URL: https://issues.apache.org/jira/browse/HBASE-15131 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Enis Soztutar > Fix For: 2.0.0 > > > Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL > by default now that it has seen some production use. > Most of the known issues has been fixed I believe for replication, metrics > etc. See HBASE-14457. > Using bounded provider with default 4 (assuming 12 disks). Should we also > look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974036#comment-15974036
]
Hudson commented on HBASE-17925:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2884 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2884/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
719a30b11a7fddc5f8e5318064d8fb9ef5eab7cb)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16215) clean up references for EOM release lines
[ https://issues.apache.org/jira/browse/HBASE-16215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974035#comment-15974035 ] Hudson commented on HBASE-16215: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2884 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2884/]) HBASE-16215 clean up of ref guide and site for EOM versions. (busbey: rev a8e6f33791d787eaf9c1bfe63aa7f3266f25268b) * (edit) src/main/asciidoc/_chapters/configuration.adoc * (edit) src/main/asciidoc/_chapters/upgrading.adoc * (edit) src/main/asciidoc/_chapters/developer.adoc * (edit) src/main/asciidoc/_chapters/cp.adoc * (edit) src/main/site/site.xml * (edit) src/main/asciidoc/_chapters/community.adoc > clean up references for EOM release lines > - > > Key: HBASE-16215 > URL: https://issues.apache.org/jira/browse/HBASE-16215 > Project: HBase > Issue Type: Task > Components: community, website >Reporter: Sean Busbey >Assignee: Sean Busbey > Attachments: HBASE-16215.0.patch, HBASE-16215.1.patch > > > I've seen us state a few places that 1.0.z is EOM. We should clean up > remaining references > * remove 1.0.z artifact from dist.apache > * remove it from the ref guide (java prereqs, hadoop prereqs, RM list) > * remove unreleased 1.0.z versions from JIRA > * archive released 1.0.z versions in JIRA > edit: expanded to include > * 0.94.y [first EOM > DISCUSS|https://lists.apache.org/thread.html/547058fc59cb48130b35c80f86d41c28038195c99f5ed94e834e291c@%3Cdev.hbase.apache.org%3E], > [second EOM > DISCUSS|https://lists.apache.org/thread.html/5ed4ae42ee9be61f963dd0a49f6896fb7771cd8d319d2d86b3c3cc76@%3Cdev.hbase.apache.org%3E] > * 0.98.y [EOM > DISCUSS|https://lists.apache.org/thread.html/fa968458e0f7f224cf906a259448a207b1230548ce6f2c0d50ceeb12@%3Cdev.hbase.apache.org%3E] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17929) Add more options for PE tool
[ https://issues.apache.org/jira/browse/HBASE-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-17929. --- Resolution: Fixed Pushed the addendum to master. > Add more options for PE tool > > > Key: HBASE-17929 > URL: https://issues.apache.org/jira/browse/HBASE-17929 > Project: HBase > Issue Type: Sub-task > Components: Performance, test >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17929-addendum.patch, HBASE-17929.patch > > > For better evaluating scan performance. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default
[ https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974029#comment-15974029 ] Duo Zhang commented on HBASE-15131: --- Yeah this could be done together with the async wal test. > Turn on multi-WAL by default > > > Key: HBASE-15131 > URL: https://issues.apache.org/jira/browse/HBASE-15131 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Enis Soztutar > Fix For: 2.0.0 > > > Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL > by default now that it has seen some production use. > Most of the known issues has been fixed I believe for replication, metrics > etc. See HBASE-14457. > Using bounded provider with default 4 (assuming 12 disks). Should we also > look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17929) Add more options for PE tool
[ https://issues.apache.org/jira/browse/HBASE-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17929: -- Attachment: HBASE-17929-addendum.patch > Add more options for PE tool > > > Key: HBASE-17929 > URL: https://issues.apache.org/jira/browse/HBASE-17929 > Project: HBase > Issue Type: Sub-task > Components: Performance, test >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17929-addendum.patch, HBASE-17929.patch > > > For better evaluating scan performance. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (HBASE-17929) Add more options for PE tool
[ https://issues.apache.org/jira/browse/HBASE-17929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-17929: --- Forget to add scan read type support. > Add more options for PE tool > > > Key: HBASE-17929 > URL: https://issues.apache.org/jira/browse/HBASE-17929 > Project: HBase > Issue Type: Sub-task > Components: Performance, test >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17929.patch > > > For better evaluating scan performance. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default
[ https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974017#comment-15974017 ] stack commented on HBASE-15131: --- Oh, you want to see if this will help you... Maybe [~busbey] has his more recent results? > Turn on multi-WAL by default > > > Key: HBASE-15131 > URL: https://issues.apache.org/jira/browse/HBASE-15131 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Enis Soztutar > Fix For: 2.0.0 > > > Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL > by default now that it has seen some production use. > Most of the known issues has been fixed I believe for replication, metrics > etc. See HBASE-14457. > Using bounded provider with default 4 (assuming 12 disks). Should we also > look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default
[ https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974016#comment-15974016 ] stack commented on HBASE-15131: --- Not sure [~carp84] I'd rather work on the async dfs client-based WAL implementaiton try and figure why that isn't going fast First. > Turn on multi-WAL by default > > > Key: HBASE-15131 > URL: https://issues.apache.org/jira/browse/HBASE-15131 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Enis Soztutar > Fix For: 2.0.0 > > > Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL > by default now that it has seen some production use. > Most of the known issues has been fixed I believe for replication, metrics > etc. See HBASE-14457. > Using bounded provider with default 4 (assuming 12 disks). Should we also > look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17675) ReplicationEndpoint should choose new sinks if a SaslException occurs
[ https://issues.apache.org/jira/browse/HBASE-17675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-17675: - Summary: ReplicationEndpoint should choose new sinks if a SaslException occurs (was: ReplicationEndpoint should choose new sinks if a SaslException occurs ) > ReplicationEndpoint should choose new sinks if a SaslException occurs > - > > Key: HBASE-17675 > URL: https://issues.apache.org/jira/browse/HBASE-17675 > Project: HBase > Issue Type: Bug >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.10 > > Attachments: HBASE-17675.patch > > > We had an issue where a regionserver on our destination side failed to > refresh the keytabs. The source side's replication got stuck because the > HBaseInterClusterReplicationEndpoint only chooses new sinks if there happens > to be a ConnectException but the SaslException is an IOException, which does > not choose new sinks. > I'll put up a patch to check this exception and choose new sinks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17667) Implement async flush/compact region methods
[
https://issues.apache.org/jira/browse/HBASE-17667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-17667:
-
Attachment: HBASE-17667.v2.patch
Patch v2:
1. split CompletableFuture compactRegionServer(ServerName sn, boolean
major) into two method :
CompletableFuture compactRegionServer(ServerName sn) ;
CompletableFuture majorCompactRegionServer(ServerName sn) ;
2. add a UT for both compactRegionServer and majorCompactRegionServer.
> Implement async flush/compact region methods
> --
>
> Key: HBASE-17667
> URL: https://issues.apache.org/jira/browse/HBASE-17667
> Project: HBase
> Issue Type: Sub-task
> Components: Admin, asyncclient, Client
>Affects Versions: 2.0.0
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-17667.v1.patch, HBASE-17667.v1.patch,
> HBASE-17667.v2.patch
>
>
> Implement following methods for async admin:
> {code}
> 1. flush table ;
> 2. flush region;
> 3. compact table;
> 4. compact region;
> 5. compact region server;
> 6. major compact for table;
> 7. major compact for region;
> 8. major compact for CF;
> 9. major compact for specific region and specific CF;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17513) Thrift Server 1 uses different QOP settings than RPC and Thrift Server 2 and can easily be misconfigured so there is no encryption when the operator expects it.
[
https://issues.apache.org/jira/browse/HBASE-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-17513:
-
Fix Version/s: (was: 1.1.10)
1.1.11
> Thrift Server 1 uses different QOP settings than RPC and Thrift Server 2 and
> can easily be misconfigured so there is no encryption when the operator
> expects it.
>
>
> Key: HBASE-17513
> URL: https://issues.apache.org/jira/browse/HBASE-17513
> Project: HBase
> Issue Type: Bug
> Components: documentation, security, Thrift, Usability
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0, 1.2.6, 1.3.2, 1.1.11
>
>
> As of HBASE-14400 the setting {{hbase.thrift.security.qop}} was unified to
> behave the same as the general HBase RPC protection. However, this only
> happened for the Thrift2 server. The Thrift server found in the thrift
> package (aka Thrift Server 1) still hard codes the old configs of 'auth',
> 'auth-int', and 'auth-conf'.
> Additionally, these Quality of Protection (qop) settings are used only by the
> SASL transport. If a user configures the HBase Thrift Server to make use of
> the HTTP transport (to enable doAs proxying e.g. for Hue) then a QOP setting
> of 'privacy' or 'auth-conf' won't get them encryption as expected.
> We should
> 1) update {{hbase-thrift/src/main/.../thrift/ThriftServerRunner}} to rely on
> {{SaslUtil}} to use the same 'authentication', 'integrity', 'privacy' configs
> in a backward compatible way
> 2) also have ThriftServerRunner warn when both {{hbase.thrift.security.qop}}
> and {{hbase.regionserver.thrift.http}} are set, since the latter will cause
> the former to be ignored. (users should be directed to
> {{hbase.thrift.ssl.enabled}} and related configs to ensure their transport is
> encrypted when using the HTTP transport.)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-17885: - Fix Version/s: (was: 1.1.9) 1.1.11 > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 1.2.6, 1.3.2, 1.1.11 > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default
[ https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974006#comment-15974006 ] Yu Li commented on HBASE-15131: --- Should we revive this thread for 2.0.0? [~stack] [~Apache9] could you also take a look here? Thanks. > Turn on multi-WAL by default > > > Key: HBASE-15131 > URL: https://issues.apache.org/jira/browse/HBASE-15131 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Enis Soztutar > Fix For: 2.0.0 > > > Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL > by default now that it has seen some production use. > Most of the known issues has been fixed I believe for replication, metrics > etc. See HBASE-14457. > Using bounded provider with default 4 (assuming 12 disks). Should we also > look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-17534: - Fix Version/s: (was: 1.1.10) 1.1.11 > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973981#comment-15973981
]
Hudson commented on HBASE-17925:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #123 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/123/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
d0d162d1f7594ef0454d0aed60c391f268e68dbd)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973971#comment-15973971
]
Hudson commented on HBASE-17925:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #119 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/119/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
d0d162d1f7594ef0454d0aed60c391f268e68dbd)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973965#comment-15973965
]
Hudson commented on HBASE-17925:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #146 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/146/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
acbba3749ad02c62ae4b45e7ba07321ae9400d8a)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17915) Implement async replication admin methods
[ https://issues.apache.org/jira/browse/HBASE-17915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973963#comment-15973963 ] Guanghao Zhang commented on HBASE-17915: Timeout ut not related. And I try these ut on my computer and passed locally. Will commit it if no objection. > Implement async replication admin methods > - > > Key: HBASE-17915 > URL: https://issues.apache.org/jira/browse/HBASE-17915 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17915.master.001.patch, > HBASE-17915.master.001.patch, HBASE-17915.master.001.patch, > HBASE-17915.master.002.patch, HBASE-17915.master.003.patch, > HBASE-17915.master.004.patch, HBASE-17915.master.005.patch, > HBASE-17915.master.005.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973915#comment-15973915
]
Hudson commented on HBASE-17925:
FAILURE: Integrated in Jenkins build HBase-1.4 #697 (See
[https://builds.apache.org/job/HBase-1.4/697/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
6ea6915245ed276f441a8686daf2c9a66cf9115f)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17939) Backport HBASE-17914 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17939: -- Component/s: regionserver Compaction > Backport HBASE-17914 to branch-1 > > > Key: HBASE-17939 > URL: https://issues.apache.org/jira/browse/HBASE-17939 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Duo Zhang > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17939) Backport HBASE-17914 to branch-1
Duo Zhang created HBASE-17939: - Summary: Backport HBASE-17914 to branch-1 Key: HBASE-17939 URL: https://issues.apache.org/jira/browse/HBASE-17939 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973909#comment-15973909
]
Hudson commented on HBASE-17925:
SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1857 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1857/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
c50d40a15d66959590b9ccd50667a190b3a94059)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction
[ https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17914: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~stack] and [~ram_krish] for reviewing. > Create a new reader instead of cloning a new StoreFile when compaction > -- > > Key: HBASE-17914 > URL: https://issues.apache.org/jira/browse/HBASE-17914 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17914.patch, HBASE-17914-v1.patch, > HBASE-17914-v1.patch, HBASE-17914-v2.patch, HBASE-17914-v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15296) Break out writer and reader from StoreFile
[
https://issues.apache.org/jira/browse/HBASE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973883#comment-15973883
]
Hadoop QA commented on HBASE-15296:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 12s {color}
| {color:red} HBASE-15296 does not apply to branch-1. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806778/HBASE-15296-branch-1-v1.patch
|
| JIRA Issue | HBASE-15296 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6496/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Break out writer and reader from StoreFile
> --
>
> Key: HBASE-15296
> URL: https://issues.apache.org/jira/browse/HBASE-15296
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-15296-branch-1.1.patch,
> HBASE-15296-branch-1.2.patch, HBASE-15296-branch-1.patch,
> HBASE-15296-branch-1.patch, HBASE-15296-branch-1-v1.patch,
> HBASE-15296-master.patch, HBASE-15296-master-v2.patch,
> HBASE-15296-master-v3.patch, HBASE-15296-master-v4.patch,
> HBASE-15296-master-v5.patch
>
>
> StoreFile.java is trending to become a monolithic class, it's ~1800 lines.
> Would it make sense to break out reader and writer (~500 lines each) into
> separate files.
> We are doing so many different things in a single class: comparators, reader,
> writer, other stuff; and it hurts readability a lot, to the point that just
> reading through a piece of code require scrolling up and down to see which
> level (reader/writer/base class level) it belongs to. These small-small
> things really don't help while trying to understanding the code. There are
> good reasons we don't do these often (affects existing patches, needs to be
> done for all branches, etc). But this and a few other classes can really use
> a single iteration of refactoring to make things a lot better.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16314) Retry on table snapshot failure
[
https://issues.apache.org/jira/browse/HBASE-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973880#comment-15973880
]
Hadoop QA commented on HBASE-16314:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
41m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 136m 59s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 207m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.util.TestHBaseFsckTwoRS |
| | hadoop.hbase.util.TestHBaseFsckReplicas |
| Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
| | org.apache.hadoop.hbase.regionserver.TestCompactSplitThread |
| | org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss |
| | org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| | org.apache.hadoop.hbase.regionserver.wal.TestLogRolling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863902/HBASE-16314-v5.patch |
| JIRA Issue | HBASE-16314 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux acdb02381f09 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a8e6f33 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6495/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCo
[jira] [Commented] (HBASE-15296) Break out writer and reader from StoreFile
[ https://issues.apache.org/jira/browse/HBASE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973877#comment-15973877 ] Duo Zhang commented on HBASE-15296: --- Any updates on backporting to branch-1 here? The backporting of HBASE-17914 will be much easier if we could have this patch in branch-1. Thanks. > Break out writer and reader from StoreFile > -- > > Key: HBASE-15296 > URL: https://issues.apache.org/jira/browse/HBASE-15296 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-15296-branch-1.1.patch, > HBASE-15296-branch-1.2.patch, HBASE-15296-branch-1.patch, > HBASE-15296-branch-1.patch, HBASE-15296-branch-1-v1.patch, > HBASE-15296-master.patch, HBASE-15296-master-v2.patch, > HBASE-15296-master-v3.patch, HBASE-15296-master-v4.patch, > HBASE-15296-master-v5.patch > > > StoreFile.java is trending to become a monolithic class, it's ~1800 lines. > Would it make sense to break out reader and writer (~500 lines each) into > separate files. > We are doing so many different things in a single class: comparators, reader, > writer, other stuff; and it hurts readability a lot, to the point that just > reading through a piece of code require scrolling up and down to see which > level (reader/writer/base class level) it belongs to. These small-small > things really don't help while trying to understanding the code. There are > good reasons we don't do these often (affects existing patches, needs to be > done for all branches, etc). But this and a few other classes can really use > a single iteration of refactoring to make things a lot better. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction
[ https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973872#comment-15973872 ] Duo Zhang commented on HBASE-17914: --- Backporting the patch to branch-1 is not straight-forward as StoreFileReader is still called StoreFile.reader. Will open a backporting issue to do it. > Create a new reader instead of cloning a new StoreFile when compaction > -- > > Key: HBASE-17914 > URL: https://issues.apache.org/jira/browse/HBASE-17914 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17914.patch, HBASE-17914-v1.patch, > HBASE-17914-v1.patch, HBASE-17914-v2.patch, HBASE-17914-v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973865#comment-15973865
]
Hudson commented on HBASE-17925:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #158 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/158/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
acbba3749ad02c62ae4b45e7ba07321ae9400d8a)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17924) Consider sorting the row order when processing multi() ops before taking rowlocks
[
https://issues.apache.org/jira/browse/HBASE-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973863#comment-15973863
]
Allan Yang commented on HBASE-17924:
{quote}
Test failure looks related to patch.
org.apache.hadoop.hbase.wal.WALSplitter$MutationReplay defines
compareTo(WALSplitter$MutationReplay) and uses Object.equals() At
WALSplitter.java:Object.equals() At WALSplitter.java:[line 2301]
{quote}
Yes, There is a bug in the patch, I'm working on it
> Consider sorting the row order when processing multi() ops before taking
> rowlocks
> -
>
> Key: HBASE-17924
> URL: https://issues.apache.org/jira/browse/HBASE-17924
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0, 1.1.8
>Reporter: Andrew Purtell
>Assignee: Allan Yang
> Fix For: 2.0.0
>
> Attachments: HBASE-17924.patch, HBASE-17924.v0.patch
>
>
> When processing a batch mutation, we take row locks in whatever order the
> mutations were added to the multi op by the client.
>
> {noformat}
> RSRpcServices#multi -> RSRpcServices#mutateRows -> HRegion#mutateRow ->
> HRegion#mutateRowsWithLocks -> HRegion#processRowsWithLocks
> {noformat}
> Or
> {noformat}
> RSRpcServices#multi -> RSRpcServices#doNonAtomicRegionMutation ->
> HRegion#get
> | HRegion#append
> | HRegion#increment
> | HRegionServer#doBatchOp -> HRegion#batchMutate ->
> HRegion#doMiniBatchMutation
> {noformat}
>
> multi() is fed by client APIs that accept a RowMutations object containing
> actions for multiple rows. The container for ops inside RowMutations is an
> ArrayList, which doesn't change the ordering of objects added to it. The
> protobuf implementation of the messages for multi ops do not reorder the list
> of actions. When processing multi ops we iterate over the actions in the
> order rehydrated from protobuf.
> We should discuss sorting the order of ops by row key when processing multi()
> ops before taking row locks. Does this make lock ordering more predictable
> for server side operations? Yes, but potentially surprising for the client,
> right? Is there any legitimate reason we should take locks out of row key
> sorted order because the client has structured the request as such?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973860#comment-15973860
]
Hudson commented on HBASE-17925:
FAILURE: Integrated in Jenkins build HBase-1.1-JDK8 #1941 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1941/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
c50d40a15d66959590b9ccd50667a190b3a94059)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17867) Implement async procedure RPC API(list/exec/abort/isFinished)
[ https://issues.apache.org/jira/browse/HBASE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973847#comment-15973847 ] Zheng Hu commented on HBASE-17867: -- Nice explanation for async isProcedureFinished method. UT failed because of forgetting add category annotation . Will fix it and upload a new patch. > Implement async procedure RPC API(list/exec/abort/isFinished) > - > > Key: HBASE-17867 > URL: https://issues.apache.org/jira/browse/HBASE-17867 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: 2.0.0 > > Attachments: HBASE-17867.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973828#comment-15973828
]
Hudson commented on HBASE-17925:
SUCCESS: Integrated in Jenkins build HBase-1.2-IT #858 (See
[https://builds.apache.org/job/HBase-1.2-IT/858/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
d0d162d1f7594ef0454d0aed60c391f268e68dbd)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17867) Implement async procedure RPC API(list/exec/abort/isFinished)
[ https://issues.apache.org/jira/browse/HBASE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973813#comment-15973813 ] stack commented on HBASE-17867: --- Skimmed patch. Looking good. Is failure related? > Implement async procedure RPC API(list/exec/abort/isFinished) > - > > Key: HBASE-17867 > URL: https://issues.apache.org/jira/browse/HBASE-17867 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: 2.0.0 > > Attachments: HBASE-17867.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17936) Refine sum endpoint example in ref guide
[ https://issues.apache.org/jira/browse/HBASE-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973798#comment-15973798 ] Xiang Li commented on HBASE-17936: -- Thanks [~stack] ! > Refine sum endpoint example in ref guide > > > Key: HBASE-17936 > URL: https://issues.apache.org/jira/browse/HBASE-17936 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17936.master.000.patch > > > Refine chapter 90.2, the example java code of sum endpoint of > http://hbase.apache.org/book.html#cp_example > - error > - typo > - genericity > - indent -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16465) Disable region splits and merges, balancer during full backup
[ https://issues.apache.org/jira/browse/HBASE-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16465: -- Fix Version/s: (was: HBASE-7912) 2.0.0 > Disable region splits and merges, balancer during full backup > - > > Key: HBASE-16465 > URL: https://issues.apache.org/jira/browse/HBASE-16465 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-16465-v1.patch, HBASE-16465-v2.patch, > HBASE-16465-v3.patch, HBASE-16465-v4.patch, HBASE-16465-v5.patch, > HBASE-16465-v6.patch > > > Incorporate HBASE-15128 > Balancer, catalog janitor and region normalizer should be disabled as well > during full backup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973767#comment-15973767
]
Hudson commented on HBASE-17925:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #27 (See
[https://builds.apache.org/job/HBase-1.3-IT/27/])
HBASE-17925 mvn assembly:single fails against hadoop3-alpha2 (jmhsieh: rev
acbba3749ad02c62ae4b45e7ba07321ae9400d8a)
* (edit) pom.xml
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973754#comment-15973754
]
Jonathan Hsieh commented on HBASE-17925:
Thanks for the review Ted. Committed.
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17925) mvn assembly:single fails against hadoop3-alpha2
[
https://issues.apache.org/jira/browse/HBASE-17925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-17925:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.3.2
1.2.6
1.1.9
1.4.0
2.0.0
Status: Resolved (was: Patch Available)
> mvn assembly:single fails against hadoop3-alpha2
>
>
> Key: HBASE-17925
> URL: https://issues.apache.org/jira/browse/HBASE-17925
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.4.0, 1.1.9, 1.2.6, 1.3.2
>
> Attachments: hbase-17925.patch
>
>
> generally to build tarballs against hadoop3 alpha2 we'd use
> {code}
> mvn -Dhadoop.profile=3.0 -Dhadoop.three-version=3.0.0-alpha2 assembly:single
> -DskipTests
> {code}
> This currently fails.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15227) HBase Backup Phase 3: Fault tolerance (client/server) support
[ https://issues.apache.org/jira/browse/HBASE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15227: -- Description: System must be tolerant to faults: # Backup operations MUST be atomic (no partial completion state in the backup system table) # Process must detect any type of failures which can result in a data loss (partial backup or partial restore) # Proper system table state restore and cleanup must be done in case of a failure # Additional utility to repair backup system table and corresponding file system cleanup must be implemented was: System must be tolerant to faults: # Backup operations MUST be atomic (no partial completion state in the backup system table) # Process must detect any type of failures which can result in a data loss (partial backup) in a backup destination # Proper system table state restore and cleanup must be done in case of a failure # Additional utility to repair backup system table and corresponding file system cleanup must be implemented > HBase Backup Phase 3: Fault tolerance (client/server) support > - > > Key: HBASE-15227 > URL: https://issues.apache.org/jira/browse/HBASE-15227 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15227-v3.patch, HBASE-15277-v1.patch > > > System must be tolerant to faults: > # Backup operations MUST be atomic (no partial completion state in the backup > system table) > # Process must detect any type of failures which can result in a data loss > (partial backup or partial restore) > # Proper system table state restore and cleanup must be done in case of a > failure > # Additional utility to repair backup system table and corresponding file > system cleanup must be implemented -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15227) HBase Backup Phase 3: Fault tolerance (client/server) support
[ https://issues.apache.org/jira/browse/HBASE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15227: -- Description: System must be tolerant to faults: # Backup operations MUST be atomic (no partial completion state in the backup system table) # Process must detect any type of failures which can result in a data loss (partial backup) in a backup destination # Proper system table state restore and cleanup must be done in case of a failure # Additional utility to repair backup system table and corresponding file system cleanup must be implemented was: System must be tolerant to faults: # Backup operations MUST be atomic (no partial completion state in system table) # Process must detect any type of failures which can result in a data loss (partial backup) in a backup destination # Proper system table state restore and cleanup must be done in case of a failure # Additional utility to repair backup system table and corresponding file system cleanup must be implemented > HBase Backup Phase 3: Fault tolerance (client/server) support > - > > Key: HBASE-15227 > URL: https://issues.apache.org/jira/browse/HBASE-15227 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15227-v3.patch, HBASE-15277-v1.patch > > > System must be tolerant to faults: > # Backup operations MUST be atomic (no partial completion state in the backup > system table) > # Process must detect any type of failures which can result in a data loss > (partial backup) in a backup destination > # Proper system table state restore and cleanup must be done in case of a > failure > # Additional utility to repair backup system table and corresponding file > system cleanup must be implemented -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17924) Consider sorting the row order when processing multi() ops before taking rowlocks
[
https://issues.apache.org/jira/browse/HBASE-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973693#comment-15973693
]
Enis Soztutar commented on HBASE-17924:
---
Good point, we do this sorting in MultiRowMutationEndpoint:
{code}
// set of rows to lock, sorted to avoid deadlocks
SortedSet rowsToLock = new TreeSet<>(Bytes.BYTES_COMPARATOR);
List mutateRequestList = request.getMutationRequestList();
List mutations = new ArrayList<>(mutateRequestList.size());
for (MutationProto m : mutateRequestList) {
mutations.add(ProtobufUtil.toMutation(m));
}
{code}
Makes sense to do it in the regular batch path as well.
> Consider sorting the row order when processing multi() ops before taking
> rowlocks
> -
>
> Key: HBASE-17924
> URL: https://issues.apache.org/jira/browse/HBASE-17924
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0, 1.1.8
>Reporter: Andrew Purtell
>Assignee: Allan Yang
> Fix For: 2.0.0
>
> Attachments: HBASE-17924.patch, HBASE-17924.v0.patch
>
>
> When processing a batch mutation, we take row locks in whatever order the
> mutations were added to the multi op by the client.
>
> {noformat}
> RSRpcServices#multi -> RSRpcServices#mutateRows -> HRegion#mutateRow ->
> HRegion#mutateRowsWithLocks -> HRegion#processRowsWithLocks
> {noformat}
> Or
> {noformat}
> RSRpcServices#multi -> RSRpcServices#doNonAtomicRegionMutation ->
> HRegion#get
> | HRegion#append
> | HRegion#increment
> | HRegionServer#doBatchOp -> HRegion#batchMutate ->
> HRegion#doMiniBatchMutation
> {noformat}
>
> multi() is fed by client APIs that accept a RowMutations object containing
> actions for multiple rows. The container for ops inside RowMutations is an
> ArrayList, which doesn't change the ordering of objects added to it. The
> protobuf implementation of the messages for multi ops do not reorder the list
> of actions. When processing multi ops we iterate over the actions in the
> order rehydrated from protobuf.
> We should discuss sorting the order of ops by row key when processing multi()
> ops before taking row locks. Does this make lock ordering more predictable
> for server side operations? Yes, but potentially surprising for the client,
> right? Is there any legitimate reason we should take locks out of row key
> sorted order because the client has structured the request as such?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973689#comment-15973689
]
Hadoop QA commented on HBASE-17937:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 31s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 7s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 8s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 165m 9s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 218m 58s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2
|
| | hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
| | hadoop.hbase.regionserver.TestCompactionInDeadRegionServer |
| | hadoop.hbase.snapshot.TestExportSnapshot |
| | hadoop.hbase.util.TestHBaseFsck |
| | hadoop.hbase.client.TestClientOperationInterrupt |
| | hadoop.hbase.master.TestMasterBalanceThrottling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863870/HBASE-17937.branch-1.001.patch
|
| JIRA Issue | HBASE-17937 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ed2ee2b8f7ad 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 / c8c2e07 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6491/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6491/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6491/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test
[jira] [Commented] (HBASE-17867) Implement async procedure RPC API(list/exec/abort/isFinished)
[ https://issues.apache.org/jira/browse/HBASE-17867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973688#comment-15973688 ] stack commented on HBASE-17867: --- Why not an async isProcedureFinished? It involves a trip to the Master, Master does internal query, perhaps a GC hits at this exact time Better if we don't tie up the client while all this work is going on? > Implement async procedure RPC API(list/exec/abort/isFinished) > - > > Key: HBASE-17867 > URL: https://issues.apache.org/jira/browse/HBASE-17867 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Zheng Hu >Assignee: Zheng Hu > Fix For: 2.0.0 > > Attachments: HBASE-17867.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16181) Allow snapshot of hbase:backup table
[
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973681#comment-15973681
]
Hadoop QA commented on HBASE-16181:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
67m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 12s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 50s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 105m 13s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863889/HBASE-16181-v3.patch |
| JIRA Issue | HBASE-16181 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8f034b45a94a 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d15f75b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6493/testReport/ |
| modules | C: hbase-common hbase-client U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6493/console |
| Powered by | Apache Yetu
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973674#comment-15973674
]
Enis Soztutar commented on HBASE-17937:
---
Makes sense.
For the patch, you can remove commented-out code:
{code}
// TEST_UTIL.getConfiguration().setLong(HConstants.HREGION_MEMSTORE_FLUSH_SIZE,
100);
{code}
Also rename the {{SlowRegionObserver}}, since it is not a slow coprocessor, but
instead triggers flush itself.
> Memstore size becomes negative in case of expensive postPut/Delete
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch,
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch,
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update
> the memstore size in the finally block of doMiniBatchMutate, however a queued
> flush can be triggered during the coprocessor calls(if they are taking time
> eg. index updates) since we have released the locks and advanced mvcc at this
> point. The flush will turn the memstore size negative since the value
> subtracted is the actual value flushed from stores. The negative value
> impacts the future flushes amongst others that depend on memstore size.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17914) Create a new reader instead of cloning a new StoreFile when compaction
[ https://issues.apache.org/jira/browse/HBASE-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973667#comment-15973667 ] stack commented on HBASE-17914: --- +1 Lets try it. > Create a new reader instead of cloning a new StoreFile when compaction > -- > > Key: HBASE-17914 > URL: https://issues.apache.org/jira/browse/HBASE-17914 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17914.patch, HBASE-17914-v1.patch, > HBASE-17914-v1.patch, HBASE-17914-v2.patch, HBASE-17914-v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16181) Allow snapshot of hbase:backup table
[
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973635#comment-15973635
]
Hadoop QA commented on HBASE-16181:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 6s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 13s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 34s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863896/HBASE-16181-v4.patch |
| JIRA Issue | HBASE-16181 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5aad303b2ec9 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d15f75b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6494/testReport/ |
| modules | C: hbase-common hbase-client U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6494/console |
| Powered by | Apach
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973624#comment-15973624
]
Hadoop QA commented on HBASE-17937:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 12s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 112m 40s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 45s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | hadoop.hbase.snapshot.TestSecureExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863873/HBASE-17937.master.002.patch
|
| JIRA Issue | HBASE-17937 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 793e72be90b3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d15f75b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6492/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6492/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6492/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6492/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Memstore size becomes negative in case of expensive postPut/Del
[jira] [Updated] (HBASE-16314) Retry on table snapshot failure
[ https://issues.apache.org/jira/browse/HBASE-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16314: -- Attachment: HBASE-16314-v5.patch v5. Rebased to current master. cc: [~tedyu] > Retry on table snapshot failure > --- > > Key: HBASE-16314 > URL: https://issues.apache.org/jira/browse/HBASE-16314 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-16314-v1.patch, HBASE-16314-v2.patch, > HBASE-16314-v3.patch, HBASE-16314-v4.patch, HBASE-16314-v5.patch > > > Table snapshot (during full backup) can fail with NotServingRegionException > (splits, AM region moves). We need to add retry logic here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973578#comment-15973578 ] Steve Loughran commented on HBASE-17877: bq Also while looking at some UnsignedBytes function for the guava version we were using I noticed that guava v14.0.1 also uses this implementation so probably hadoop borrowed it from there .. I was about to say "no it doesn't", but then I saw the comment at the top, "This is borrowed and slightly modified from Guava's". Been in there a long time though (since 2012), so it's history is lost. x86 pert is what really matters, though we don't want to be pathologically antisocial to the other CPU arches, not just for the sake of PPC, but for when Arm goes mainstream in the DC. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16181) Allow snapshot of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16181: -- Attachment: HBASE-16181-v4.patch v4. Do we do any security checks during snapshots? If, yes, then we are fine - only super user will be able to run snapshot on a system table. > Allow snapshot of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch, > HBASE-16181-v3.patch, HBASE-16181-v4.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16181) Allow snapshot of hbase:backup table
[
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973545#comment-15973545
]
Ted Yu commented on HBASE-16181:
{code}
536 public void setSnapshottable(boolean value) {
{code}
Returning TableName (this) would allow chaining.
Is there any security concern with this new API ?
> Allow snapshot of hbase:backup table
>
>
> Key: HBASE-16181
> URL: https://issues.apache.org/jira/browse/HBASE-16181
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch,
> HBASE-16181-v3.patch
>
>
> Snapshot of HBase system tables is not supported, we need either move
> hbase:backup into different name space or fix snapshots.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17935) NPE when using PreemptiveFastFailInterceptor with HBaseAdmin client
[ https://issues.apache.org/jira/browse/HBASE-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973528#comment-15973528 ] Ted Yu commented on HBASE-17935: For unit test, refer to TestFastFail where HBASE_CLIENT_FAST_FAIL_MODE_ENABLED is set true. See if namespaceExists() triggers the NPE. > NPE when using PreemptiveFastFailInterceptor with HBaseAdmin client > --- > > Key: HBASE-17935 > URL: https://issues.apache.org/jira/browse/HBASE-17935 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.1.2 >Reporter: Robert Yokota >Assignee: Robert Yokota >Priority: Minor > Attachments: HBASE-17935.master.001.patch > > > When setting hbase.client.fast.fail.mode.enabled=true and using the admin > client, got a NPE: > java.lang.NullPointerException > at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) > at > org.apache.hadoop.hbase.client.PreemptiveFastFailInterceptor.intercept(PreemptiveFastFailInterceptor.java:115) > at > org.apache.hadoop.hbase.client.PreemptiveFastFailInterceptor.intercept(PreemptiveFastFailInterceptor.java:376) > at > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:135) > at > org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297) > at > org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289) > at > org.apache.hadoop.hbase.client.HBaseAdmin.getNamespaceDescriptor(HBaseAdmin.java:3009) > at > com.yammer.calmie.store.hbase.HBaseConnectionFactory.namespaceExists(HBaseConnectionFactory.java:146) > at > com.yammer.calmie.store.hbase.HBaseConnectionFactory.createHTable(HBaseConnectionFactory.java:116) > at > com.yammer.calmie.store.hbase.HBaseConnectionFactory.declareTable(HBaseConnectionFactory.java:97) > at > com.yammer.feedie.core.index.calmie.MessageCalmieIndex.(MessageCalmieIndex.java:46) > at com.yammer.feedie.core.MessageStore.init(MessageStore.java:108) > at com.yammer.feedie.core.MessageStore.(MessageStore.java:89) > at com.yammer.feedie.FeedieService.setup(FeedieService.java:148) > at com.yammer.feedie.FeedieService.run(FeedieService.java:315) > at com.yammer.feedie.FeedieService.run(FeedieService.java:65) > at > com.yammer.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:38) > at > com.yammer.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:58) > at com.yammer.dropwizard.cli.Cli.run(Cli.java:53) > at com.yammer.dropwizard.Service.run(Service.java:61) > at com.yammer.feedie.FeedieService.main(FeedieService.java:84) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16314) Retry on table snapshot failure
[ https://issues.apache.org/jira/browse/HBASE-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16314: -- Labels: backup (was: ) > Retry on table snapshot failure > --- > > Key: HBASE-16314 > URL: https://issues.apache.org/jira/browse/HBASE-16314 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-16314-v1.patch, HBASE-16314-v2.patch, > HBASE-16314-v3.patch, HBASE-16314-v4.patch > > > Table snapshot (during full backup) can fail with NotServingRegionException > (splits, AM region moves). We need to add retry logic here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16314) Retry on table snapshot failure
[ https://issues.apache.org/jira/browse/HBASE-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16314: -- Fix Version/s: (was: HBASE-7912) 2.0.0 > Retry on table snapshot failure > --- > > Key: HBASE-16314 > URL: https://issues.apache.org/jira/browse/HBASE-16314 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16314-v1.patch, HBASE-16314-v2.patch, > HBASE-16314-v3.patch, HBASE-16314-v4.patch > > > Table snapshot (during full backup) can fail with NotServingRegionException > (splits, AM region moves). We need to add retry logic here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16181) Allow snapshot of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973521#comment-15973521 ] Vladimir Rodionov commented on HBASE-16181: --- cc: [~tedyu], [~chenheng] > Allow snapshot of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch, > HBASE-16181-v3.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16181) Allow snapshot of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16181: -- Status: Patch Available (was: Reopened) > Allow snapshot of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch, > HBASE-16181-v3.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16181) Allow snapshot of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16181: -- Attachment: HBASE-16181-v3.patch v3. Introduces TableName.isSnapshottable/ TableName.setSnapshottable By default, system tables are not snapshottable, but it can be overridden with the new API. No more checks on system tables during restore from snapshots. As since, new functionality is not exposed to command line tools, there is no way to create snapshots for system tables using command line tools. > Allow snapshot of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-16181-v1.patch, HBASE-16181-v2.patch, > HBASE-16181-v3.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973451#comment-15973451
]
Hadoop QA commented on HBASE-17937:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
57m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 55s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863866/HBASE-17937.master.002.patch
|
| JIRA Issue | HBASE-17937 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f6009d4a7afa 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d15f75b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6490/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6490/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6490/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6490/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Memstore size becomes negative in case of expensive postPut/Delete
> Coprocessor call
> ---
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973407#comment-15973407 ] Lars Hofhansl commented on HBASE-17937: --- +1 on branch-1 patch. (nice to get rid of the call in the finally block!) For the master patch, shouldn't we also move the size update after the sync of the wal? [~Apache9], [~abhishek.chouhan] > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, > HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[
https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973398#comment-15973398
]
Ted Yu commented on HBASE-17937:
lgtm
nit:
{code}
+ threwIOE = true;
{code}
Rename variable IOEThrown
> Memstore size becomes negative in case of expensive postPut/Delete
> Coprocessor call
> ---
>
> Key: HBASE-17937
> URL: https://issues.apache.org/jira/browse/HBASE-17937
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.1, 0.98.24
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-17937.branch-1.001.patch,
> HBASE-17937.master.001.patch, HBASE-17937.master.002.patch,
> HBASE-17937.master.002.patch
>
>
> We ran into a situation where the memstore size became negative due to
> expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update
> the memstore size in the finally block of doMiniBatchMutate, however a queued
> flush can be triggered during the coprocessor calls(if they are taking time
> eg. index updates) since we have released the locks and advanced mvcc at this
> point. The flush will turn the memstore size negative since the value
> subtracted is the actual value flushed from stores. The negative value
> impacts the future flushes amongst others that depend on memstore size.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973367#comment-15973367 ] churro morales commented on HBASE-17937: correct, and if you can't flush the memstore then you could potentially get into a state where you are waiting trying to allocate space in the MSlab which is done under row locks. > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, > HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-17937: --- Attachment: HBASE-17937.master.002.patch Added category annotation to the test. > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch, > HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973339#comment-15973339 ] Abhishek Singh Chouhan commented on HBASE-17937: In the patch for branch-1, have added the this.addAndGetGlobalMemstoreSize(addedSize) call just after step 7 (wal sync) and removed the update from finally block. After step 7 we are not going to rollback the memstore (we set doRollbackMemstore=false), hence we can update the memstore size there, before step 7 we don't want to update the memstore size anyways since we'll do a rollback of the memstore in the finally block. [~Apache9] [~tedyu] [~apurtell] [~lhofhansl] > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-17937: --- Attachment: HBASE-17937.branch-1.001.patch > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.branch-1.001.patch, > HBASE-17937.master.001.patch, HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17938) General fault - tolerance framework for backup/restore operations
Vladimir Rodionov created HBASE-17938: - Summary: General fault - tolerance framework for backup/restore operations Key: HBASE-17938 URL: https://issues.apache.org/jira/browse/HBASE-17938 Project: HBase Issue Type: Sub-task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov The framework must take care of all general types of failures during backup/ restore and restore system to the original state in case of a failure. That won't solve all the possible issues but we have a separate JIRAs for them as a sub-tasks of HBASE-15277 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-15226) HBase Backup/Restore Phase 3: Add FSM to Backup state
[ https://issues.apache.org/jira/browse/HBASE-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-15226. --- Resolution: Later > HBase Backup/Restore Phase 3: Add FSM to Backup state > - > > Key: HBASE-15226 > URL: https://issues.apache.org/jira/browse/HBASE-15226 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: HBASE-7912 > > > Finate State Machine. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17924) Consider sorting the row order when processing multi() ops before taking rowlocks
[
https://issues.apache.org/jira/browse/HBASE-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973270#comment-15973270
]
Andrew Purtell commented on HBASE-17924:
Test failure looks related to patch.
org.apache.hadoop.hbase.wal.WALSplitter$MutationReplay defines
compareTo(WALSplitter$MutationReplay) and uses Object.equals() At
WALSplitter.java:Object.equals() At WALSplitter.java:[line 2301]
> Consider sorting the row order when processing multi() ops before taking
> rowlocks
> -
>
> Key: HBASE-17924
> URL: https://issues.apache.org/jira/browse/HBASE-17924
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0, 1.1.8
>Reporter: Andrew Purtell
>Assignee: Allan Yang
> Fix For: 2.0.0
>
> Attachments: HBASE-17924.patch, HBASE-17924.v0.patch
>
>
> When processing a batch mutation, we take row locks in whatever order the
> mutations were added to the multi op by the client.
>
> {noformat}
> RSRpcServices#multi -> RSRpcServices#mutateRows -> HRegion#mutateRow ->
> HRegion#mutateRowsWithLocks -> HRegion#processRowsWithLocks
> {noformat}
> Or
> {noformat}
> RSRpcServices#multi -> RSRpcServices#doNonAtomicRegionMutation ->
> HRegion#get
> | HRegion#append
> | HRegion#increment
> | HRegionServer#doBatchOp -> HRegion#batchMutate ->
> HRegion#doMiniBatchMutation
> {noformat}
>
> multi() is fed by client APIs that accept a RowMutations object containing
> actions for multiple rows. The container for ops inside RowMutations is an
> ArrayList, which doesn't change the ordering of objects added to it. The
> protobuf implementation of the messages for multi ops do not reorder the list
> of actions. When processing multi ops we iterate over the actions in the
> order rehydrated from protobuf.
> We should discuss sorting the order of ops by row key when processing multi()
> ops before taking row locks. Does this make lock ordering more predictable
> for server side operations? Yes, but potentially surprising for the client,
> right? Is there any legitimate reason we should take locks out of row key
> sorted order because the client has structured the request as such?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-17542) Move backup system table into separate namespace
[ https://issues.apache.org/jira/browse/HBASE-17542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-17542. --- Resolution: Won't Fix Currently, the backup system table is in HBase system namespace and let it be there. > Move backup system table into separate namespace > > > Key: HBASE-17542 > URL: https://issues.apache.org/jira/browse/HBASE-17542 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: HBASE-7912 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-14440) Restore to snapshot
[ https://issues.apache.org/jira/browse/HBASE-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14440. --- Resolution: Later > Restore to snapshot > --- > > Key: HBASE-14440 > URL: https://issues.apache.org/jira/browse/HBASE-14440 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov > Labels: backup > Fix For: HBASE-7912 > > > Good to have feature: restore backup to snapshot. This will allow to massage > data with custom M/R job over snapshot. Restore time range only, or > particular key range (ranges). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973257#comment-15973257 ] Abhishek Singh Chouhan commented on HBASE-17937: Have moved the size update to be just after step 5 and removed from finally as suggested by [~Apache9] since we don't do the rollback now with the chenged mvcc. [~lhofhansl] Agreed that this is a temporary condition till the time we return from the coprocessor call and proceed to the finally part, however if we have multiple such requests that seem to take a large amount of time or are stuck, like our observation recently we can possibly run into large negatives of memstore sizes (lets say -x mb) so the next flush will happen at x+memstore flush size which will take more time and so forth. > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch, > HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated HBASE-17937: --- Attachment: HBASE-17937.master.002.patch > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch, > HBASE-17937.master.002.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973189#comment-15973189 ] Lars Hofhansl edited comment on HBASE-17937 at 4/18/17 6:06 PM: Agreed. I think it's just [~abhishek.chouhan]'s fix, with the first size update moved right after the logsync. was (Author: lhofhansl): Agreed. I think it's just [~abhishek.chouhan]'s fix, which the first size update move right after the logsync. > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973189#comment-15973189 ] Lars Hofhansl commented on HBASE-17937: --- Agreed. I think it's just [~abhishek.chouhan]'s fix, which the first size update move right after the logsync. > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973172#comment-15973172 ] Andrew Purtell commented on HBASE-17937: Let's fix this. The warning gets your attention and demands a fix, even if it's just movement of accounting updates to a better place in the code flow. > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973163#comment-15973163 ] Lars Hofhansl commented on HBASE-17937: --- So we looked at the code here. I think this is only a temporary condition like this: 1. doMiniBatchMutation starts and writes things into the memstore 2. the background flusher flushes the memstore. Decrementing the memstore size. 3. doMiniBatchMutation finishes. Incrementing the memstore size. So we only have an issue between #2 and #3. When doMiniBatchMutation finishes the size will be correct. I do _not_ think it's a big problem, just a spurious log warning. If we did not want that, we should move the incrementing of the memstore size right after the sync of the wal in doMiniBatchMutation (inside the MVCC transaction). > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17936) Refine sum endpoint example in ref guide
[ https://issues.apache.org/jira/browse/HBASE-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973151#comment-15973151 ] Hudson commented on HBASE-17936: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2882 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2882/]) HBASE-17936 Refine sum endpoint example in ref guide (stack: rev d15f75b3cfc5de4def04e94cbb965fd7f578dc34) * (edit) src/main/asciidoc/_chapters/cp.adoc > Refine sum endpoint example in ref guide > > > Key: HBASE-17936 > URL: https://issues.apache.org/jira/browse/HBASE-17936 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17936.master.000.patch > > > Refine chapter 90.2, the example java code of sum endpoint of > http://hbase.apache.org/book.html#cp_example > - error > - typo > - genericity > - indent -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973144#comment-15973144 ] Ted Yu commented on HBASE-16438: Mind highlighting how you fixed the tests ? Thanks > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_addendum.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16438: --- Status: Open (was: Patch Available) > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_addendum.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16438: --- Status: Patch Available (was: Open) > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_addendum.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16438: --- Attachment: HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch Resubmitting for QA. I have run all the test cases in the local linux box. The large tests are in progress. By the time will submit for one QA run also. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_12_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_13_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_14_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_addendum.patch, HBASE-16438.patch, > MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17937) Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
[ https://issues.apache.org/jira/browse/HBASE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973092#comment-15973092 ] churro morales commented on HBASE-17937: if you get a negative memstore size the problem is you can't flush or close the region. Is there any code path where after a negative memstore size a flush is possible? > Memstore size becomes negative in case of expensive postPut/Delete > Coprocessor call > --- > > Key: HBASE-17937 > URL: https://issues.apache.org/jira/browse/HBASE-17937 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.1, 0.98.24 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Attachments: HBASE-17937.master.001.patch > > > We ran into a situation where the memstore size became negative due to > expensive postPut/Delete Coprocessor calls in doMiniBatchMutate. We update > the memstore size in the finally block of doMiniBatchMutate, however a queued > flush can be triggered during the coprocessor calls(if they are taking time > eg. index updates) since we have released the locks and advanced mvcc at this > point. The flush will turn the memstore size negative since the value > subtracted is the actual value flushed from stores. The negative value > impacts the future flushes amongst others that depend on memstore size. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17936) Refine sum endpoint example in ref guide
[ https://issues.apache.org/jira/browse/HBASE-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-17936. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Pushed. Thank you for fixup. > Refine sum endpoint example in ref guide > > > Key: HBASE-17936 > URL: https://issues.apache.org/jira/browse/HBASE-17936 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17936.master.000.patch > > > Refine chapter 90.2, the example java code of sum endpoint of > http://hbase.apache.org/book.html#cp_example > - error > - typo > - genericity > - indent -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973007#comment-15973007 ] stack commented on HBASE-17877: --- +1 > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15583) Any HTableDescriptor we give out should be immutable
[
https://issues.apache.org/jira/browse/HBASE-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972957#comment-15972957
]
Hadoop QA commented on HBASE-15583:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 11s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 11s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 21 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
45s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 22s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
39s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
56m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 5s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 126m 18s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 14s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 23s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 33s
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 187m 7s {color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 2m 41s
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:bla
