[jira] [Comment Edited] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652990#comment-16652990 ] Jingyun Tian edited comment on HBASE-21322 at 10/17/18 5:48 AM: The whole test process is like this: 1. kill one RS 2. delete all MasterProcWALs immediately. 3. check if the cluster can fail over. But the result is there are splitting logs on HDFS, but no SeverCrashProcedure scheduled. !Screenshot from 2018-10-17 13-38-41.png! Thus some regions never assign again. These regions' state are still the same as the moment I delete all MasterProcWALs. !Screenshot from 2018-10-17 13-35-58.png! Thus some regions are recorded OPEN at a dead RS. !Screenshot from 2018-10-17 13-47-06.png! was (Author: tianjingyun): The whole test process is like this: 1. kill one RS 2. delete all MasterProcWALs immediately. 3. check if the cluster can fail over. But the result is there are splitting logs on HDFS, but no SeverCrashProcedure scheduled. !Screenshot from 2018-10-17 13-38-41.png! Thus some regions never assign again. These regions' state are still the same as the moment I delete all MasterProcWALs. !Screenshot from 2018-10-17 13-35-58.png! > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-35-58.png, Screenshot from > 2018-10-17 13-38-41.png, Screenshot from 2018-10-17 13-47-06.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652990#comment-16652990 ] Jingyun Tian edited comment on HBASE-21322 at 10/17/18 5:48 AM: [~stack] The whole test process is like this: 1. kill one RS 2. delete all MasterProcWALs immediately. 3. check if the cluster can fail over. But the result is there are splitting logs on HDFS, but no SeverCrashProcedure scheduled. !Screenshot from 2018-10-17 13-38-41.png! Thus some regions never assign again. These regions' state are still the same as the moment I delete all MasterProcWALs. !Screenshot from 2018-10-17 13-35-58.png! Thus some regions are recorded OPEN at a dead RS. !Screenshot from 2018-10-17 13-47-06.png! was (Author: tianjingyun): The whole test process is like this: 1. kill one RS 2. delete all MasterProcWALs immediately. 3. check if the cluster can fail over. But the result is there are splitting logs on HDFS, but no SeverCrashProcedure scheduled. !Screenshot from 2018-10-17 13-38-41.png! Thus some regions never assign again. These regions' state are still the same as the moment I delete all MasterProcWALs. !Screenshot from 2018-10-17 13-35-58.png! Thus some regions are recorded OPEN at a dead RS. !Screenshot from 2018-10-17 13-47-06.png! > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-35-58.png, Screenshot from > 2018-10-17 13-38-41.png, Screenshot from 2018-10-17 13-47-06.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Attachment: Screenshot from 2018-10-17 13-47-06.png > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-35-58.png, Screenshot from > 2018-10-17 13-38-41.png, Screenshot from 2018-10-17 13-47-06.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Attachment: (was: Screenshot from 2018-10-17 13-43-17.png) > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-35-58.png, Screenshot from > 2018-10-17 13-38-41.png, Screenshot from 2018-10-17 13-47-06.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Attachment: Screenshot from 2018-10-17 13-43-17.png > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-35-58.png, Screenshot from > 2018-10-17 13-38-41.png, Screenshot from 2018-10-17 13-47-06.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Attachment: (was: Screenshot from 2018-10-17 13-43-17.png) > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-35-58.png, Screenshot from > 2018-10-17 13-38-41.png, Screenshot from 2018-10-17 13-47-06.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Attachment: Screenshot from 2018-10-17 13-43-17.png > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-35-58.png, Screenshot from > 2018-10-17 13-38-41.png, Screenshot from 2018-10-17 13-43-17.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Attachment: Screenshot from 2018-10-17 13-35-58.png > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-35-58.png, Screenshot from > 2018-10-17 13-38-41.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21275) Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http server (branch 1 only)
[ https://issues.apache.org/jira/browse/HBASE-21275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652989#comment-16652989 ] stack commented on HBASE-21275: --- I pushed on branch-1. Does not go into branch-1.4 [~wchevreuil]. Mind making a patch (Usually its not this painful!). Thank you sir. You want it to go back to 1.2? Otherwise, 1.4 should be enough. Thanks sir. > Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http > server (branch 1 only) > > > Key: HBASE-21275 > URL: https://issues.apache.org/jira/browse/HBASE-21275 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 1.4.8, 1.2.7 > > Attachments: HBASE-21275-branch-1.001.patch, > HBASE-21275-branch-1.2.001.patch, HBASE-21275-branch-1.2.002.patch, > HBASE-21275-branch-1.2.003.patch, HBASE-21275-branch-1.2.003.patch > > > There's been a reasonable number of users running thrift http server on hbase > 1.x suffering with security audit tests pointing thrift server allows TRACE > requests. > After doing some search, I can see HBASE-20406 added restrictions for > TRACE/OPTIONS method when Thrift is running over http, but it relies on many > other commits applied to thrift http server. This patch was later reverted > from master. Then again later, HBASE-20004 had made TRACE/OPTIONS > configurable via "*hbase.thrift.http.allow.options.method*" property, with > both methods being disabled by default. This also seems to rely on many > changes applied to thrift http server, and a branch 1 compatible patch does > not seem feasible. > A solution for branch 1 is pretty simple though, am proposing a patch that > simply uses *WebAppContext*, instead of *Context*, as the context for the > *HttpServer* instance. *WebAppContext* will already restrict TRACE methods by > default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652990#comment-16652990 ] Jingyun Tian commented on HBASE-21322: -- The whole test process is like this: 1. kill one RS 2. delete all MasterProcWALs immediately. 3. check if the cluster can fail over. But the result is there are splitting logs on HDFS, but no SeverCrashProcedure scheduled. !Screenshot from 2018-10-17 13-38-41.png! Thus some regions never assign again. These regions' state are still the same as the moment I delete all MasterProcWALs. !Screenshot from 2018-10-17 13-35-58.png! > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-35-58.png, Screenshot from > 2018-10-17 13-38-41.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Attachment: Screenshot from 2018-10-17 13-38-41.png > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-17 13-38-41.png > > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21269) Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state in RegionStates when assign/unassign"
[ https://issues.apache.org/jira/browse/HBASE-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652980#comment-16652980 ] stack commented on HBASE-21269: --- [~Apache9] Question for you sir if you have a sec up on rb... Its about making Procedure async. Thanks. > Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state > in RegionStates when assign/unassign" > --- > > Key: HBASE-21269 > URL: https://issues.apache.org/jira/browse/HBASE-21269 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21269.branch-2.001.patch, > HBASE-21269.master.001.patch, HBASE-21269.master.003.patch > > > A bunch of this patch does not apply to branch-2 and master now we don't have > AP or UP anymore. Need to figure if we need override in branch-2 and master. > Let me upload the forward-port done so far. Can finish this when move to > branch-2.2 exercise. FYI [~Apache9] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21321) Backport HBASE-21278 to branch-2.1 and branch-2.0
[ https://issues.apache.org/jira/browse/HBASE-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21321: -- Fix Version/s: 2.1.1 2.03 > Backport HBASE-21278 to branch-2.1 and branch-2.0 > - > > Key: HBASE-21321 > URL: https://issues.apache.org/jira/browse/HBASE-21321 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.1.1, 2.0.3 > > > As the assign\unassign part is a bit different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21321) Backport HBASE-21278 to branch-2.1 and branch-2.0
[ https://issues.apache.org/jira/browse/HBASE-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652978#comment-16652978 ] stack commented on HBASE-21321: --- Thanks [~Apache9] (You can ignore my question up on the source issue -- smile -- just saw this). > Backport HBASE-21278 to branch-2.1 and branch-2.0 > - > > Key: HBASE-21321 > URL: https://issues.apache.org/jira/browse/HBASE-21321 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.1.1, 2.0.3 > > > As the assign\unassign part is a bit different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21321) Backport HBASE-21278 to branch-2.1 and branch-2.0
[ https://issues.apache.org/jira/browse/HBASE-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21321: -- Fix Version/s: (was: 2.03) 2.0.3 > Backport HBASE-21278 to branch-2.1 and branch-2.0 > - > > Key: HBASE-21321 > URL: https://issues.apache.org/jira/browse/HBASE-21321 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.1.1, 2.0.3 > > > As the assign\unassign part is a bit different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21321) Backport HBASE-21278 to branch-2.1 and branch-2.0
[ https://issues.apache.org/jira/browse/HBASE-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21321: -- Priority: Critical (was: Major) > Backport HBASE-21278 to branch-2.1 and branch-2.0 > - > > Key: HBASE-21321 > URL: https://issues.apache.org/jira/browse/HBASE-21321 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Critical > > As the assign\unassign part is a bit different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21321) Backport HBASE-21278 to branch-2.1 and branch-2.0
[ https://issues.apache.org/jira/browse/HBASE-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-21321: - Assignee: stack > Backport HBASE-21278 to branch-2.1 and branch-2.0 > - > > Key: HBASE-21321 > URL: https://issues.apache.org/jira/browse/HBASE-21321 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > > As the assign\unassign part is a bit different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652977#comment-16652977 ] stack commented on HBASE-21256: --- This is a nice patch. Look forward to trying it out. > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21256-branch-1.patch, HBASE-21256-v1.patch, > HBASE-21256-v10.patch, HBASE-21256-v11.patch, HBASE-21256-v12.patch, > HBASE-21256-v2.patch, HBASE-21256-v3.patch, HBASE-21256-v4.patch, > ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21328) why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default?
Nick.han created HBASE-21328: Summary: why HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default? Key: HBASE-21328 URL: https://issues.apache.org/jira/browse/HBASE-21328 Project: HBase Issue Type: Improvement Reporter: Nick.han hi,all I got a problem while I using hbase3.0.0-snapshot and hadoop 2.7.5 to build a hbase cluster,the problem is hbase using javax.servlet-api-3.1.0-jar witch is conflict by servlet-api-2.5.jar that in hadoop lib path, I run into hbase file and got config HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP set to false by default, this config decide whether or not include Hadoop lib to hbase class path，so the question is why we set this config to false?can we set it to true and exclude the Hadoop lib by default? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21198) Exclude dependency on net.minidev:json-smart
[
https://issues.apache.org/jira/browse/HBASE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652974#comment-16652974
]
stack commented on HBASE-21198:
---
Patch LGTM. Thanks for explanation [~dbist13]. Retrying patch overnight.
> Exclude dependency on net.minidev:json-smart
>
>
> Key: HBASE-21198
> URL: https://issues.apache.org/jira/browse/HBASE-21198
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-21198.v01.patch, HBASE-21198.v01.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14414/artifact/patchprocess/patch-javac-3.0.0.txt
> :
> {code}
> [ERROR] Failed to execute goal on project hbase-common: Could not resolve
> dependencies for project org.apache.hbase:hbase-common:jar:3.0.0-SNAPSHOT:
> Failed to collect dependencies at org.apache.hadoop:hadoop-common:jar:3.0.0
> -> org.apache.hadoop:hadoop-auth:jar:3.0.0 ->
> com.nimbusds:nimbus-jose-jwt:jar:4.41.1 ->
> net.minidev:json-smart:jar:2.3-SNAPSHOT: Failed to read artifact descriptor
> for net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact
> net.minidev:json-smart:pom:2.3-SNAPSHOT from/to dynamodb-local-oregon
> (https://s3-us-west-2.amazonaws.com/dynamodb-local/release): Access denied
> to:
> https://s3-us-west-2.amazonaws.com/dynamodb-local/release/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom
> , ReasonPhrase:Forbidden. -> [Help 1]
> {code}
> We should exclude dependency on net.minidev:json-smart
> hbase-common/bin/pom.xml has done so.
> The other pom.xml should do the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21198) Exclude dependency on net.minidev:json-smart
[
https://issues.apache.org/jira/browse/HBASE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21198:
--
Attachment: HBASE-21198.v01.patch
> Exclude dependency on net.minidev:json-smart
>
>
> Key: HBASE-21198
> URL: https://issues.apache.org/jira/browse/HBASE-21198
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-21198.v01.patch, HBASE-21198.v01.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14414/artifact/patchprocess/patch-javac-3.0.0.txt
> :
> {code}
> [ERROR] Failed to execute goal on project hbase-common: Could not resolve
> dependencies for project org.apache.hbase:hbase-common:jar:3.0.0-SNAPSHOT:
> Failed to collect dependencies at org.apache.hadoop:hadoop-common:jar:3.0.0
> -> org.apache.hadoop:hadoop-auth:jar:3.0.0 ->
> com.nimbusds:nimbus-jose-jwt:jar:4.41.1 ->
> net.minidev:json-smart:jar:2.3-SNAPSHOT: Failed to read artifact descriptor
> for net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact
> net.minidev:json-smart:pom:2.3-SNAPSHOT from/to dynamodb-local-oregon
> (https://s3-us-west-2.amazonaws.com/dynamodb-local/release): Access denied
> to:
> https://s3-us-west-2.amazonaws.com/dynamodb-local/release/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom
> , ReasonPhrase:Forbidden. -> [Help 1]
> {code}
> We should exclude dependency on net.minidev:json-smart
> hbase-common/bin/pom.xml has done so.
> The other pom.xml should do the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21278) Do not rollback successful sub procedures when rolling back a procedure
[
https://issues.apache.org/jira/browse/HBASE-21278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652972#comment-16652972
]
stack commented on HBASE-21278:
---
Should we backport a form of this to branch-2.1 [~Apache9] and adopt the
philosophy you have here regards rollback? (Though for latter, I suppose we
have to wait on 2.2Can't do it on a point release). Thanks.
> Do not rollback successful sub procedures when rolling back a procedure
> ---
>
> Key: HBASE-21278
> URL: https://issues.apache.org/jira/browse/HBASE-21278
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21278-v1.patch, HBASE-21278.patch,
> org.apache.hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure-output.txt
>
>
> https://builds.apache.org/job/HBase-Flaky-Tests/job/master/1235/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure-output.txt/*view*/
> I think the problem is
> {noformat}
> 2018-10-08 03:44:30,315 INFO [PEWorker-1]
> procedure.MasterProcedureScheduler(689): pid=43, ppid=42, state=SUCCESS,
> hasLock=false; TransitRegionStateProcedure
> table=testRollbackAndDoubleExecution,
> region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN checking lock on
> 9bac7c539ac0cff6dc5706ed375a3bfb
> 2018-10-08 03:44:30,320 ERROR [PEWorker-1] helpers.MarkerIgnoringBase(159):
> CODE-BUG: Uncaught runtime exception for pid=43, ppid=42, state=SUCCESS,
> hasLock=true; TransitRegionStateProcedure
> table=testRollbackAndDoubleExecution,
> region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN
> java.lang.UnsupportedOperationException
> at
> org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.rollbackState(TransitRegionStateProcedure.java:458)
> at
> org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.rollbackState(TransitRegionStateProcedure.java:97)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:208)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:957)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1605)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1567)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1446)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76)
> {noformat}
> Typically there is no rollback for TRSP. Need to dig more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21320) [canary] Cleanup of usage and add commentary
[ https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21320: -- Resolution: Fixed Hadoop Flags: Reviewed Release Note: Cleans up usage and docs around Canary. Does not change command-line args (though we should -- smile). Status: Resolved (was: Patch Available) Pushed to branch-2.0+ > [canary] Cleanup of usage and add commentary > > > Key: HBASE-21320 > URL: https://issues.apache.org/jira/browse/HBASE-21320 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21320.branch-2.1.001.patch, > HBASE-21320.branch-2.1.002.patch, HBASE-21320.branch-2.1.003.patch, > HBASE-21320.branch-2.1.004.patch > > > I was trying to us the Canary as a diagnosis tool figuring what was online > and what was off but I couldn't make sense of how to use it. There was a bug > where we default to regionserver 'mode' always and only way to change it was > via a system property and a cryptic class name. > Changed it so we don't specify output ahead of figuring what 'mode' we are to > run in. Cleaned-up usage and added commentary. It seems useable now (Did not > change command-line args though they need changing). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21263) Mention compression algorithm along with other storefile details
[
https://issues.apache.org/jira/browse/HBASE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652968#comment-16652968
]
Hudson commented on HBASE-21263:
Results for branch branch-2.0
[build #961 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/961/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/961//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/961//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/961//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Mention compression algorithm along with other storefile details
>
>
> Key: HBASE-21263
> URL: https://issues.apache.org/jira/browse/HBASE-21263
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Subrat Mishra
>Priority: Minor
> Labels: beginner, beginners
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21263-branch-1.patch,
> HBASE-21263-branch-2.0.patch, HBASE-21263.master.001.patch,
> HBASE-21263.master.002.patch, HBASE-21263.master.003.patch, HBASE-21263.patch
>
>
> Where we log storefile details we should also log the compression algorithm
> used to compress blocks on disk, if any.
> For example, here's a log line out of compaction:
> 2018-10-02 21:59:47,594 DEBUG
> [regionserver/host/1.1.1.1:8120-longCompactions-1538517461152]
> compactions.Compactor: Compacting
> hdfs://namenode:8020/hbase/data/default/TestTable/86037c19117a46b5b8148439ea55753b/i/3d04a7c28d6343ceb773737dbb192533,
> keycount=3335873, bloomtype=ROW, size=107.5 M, encoding=ROW_INDEX_V1,
> seqNum=154199, earliestPutTs=1538516084915
> Aside from bloom type, block encoding, and filename, it would be good to know
> compression type in this type of DEBUG or INFO level logging. A minor
> omission of information that could be helpful during debugging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21320) [canary] Cleanup of usage and add commentary
[ https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652960#comment-16652960 ] stack commented on HBASE-21320: --- The two tests are failing on this branch with the last few builds. Unrelated. Will take a look at them in another issue. > [canary] Cleanup of usage and add commentary > > > Key: HBASE-21320 > URL: https://issues.apache.org/jira/browse/HBASE-21320 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21320.branch-2.1.001.patch, > HBASE-21320.branch-2.1.002.patch, HBASE-21320.branch-2.1.003.patch, > HBASE-21320.branch-2.1.004.patch > > > I was trying to us the Canary as a diagnosis tool figuring what was online > and what was off but I couldn't make sense of how to use it. There was a bug > where we default to regionserver 'mode' always and only way to change it was > via a system property and a cryptic class name. > Changed it so we don't specify output ahead of figuring what 'mode' we are to > run in. Cleaned-up usage and added commentary. It seems useable now (Did not > change command-line args though they need changing). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21326) [amv2] Worker terminating UNNATURALLY null java.lang.ArrayIndexOutOfBoundsException: 1
[
https://issues.apache.org/jira/browse/HBASE-21326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-21326.
---
Resolution: Duplicate
Resolving as duplicate of HBASE-20973 as per [~allan163]
> [amv2] Worker terminating UNNATURALLY null
> java.lang.ArrayIndexOutOfBoundsException: 1
> --
>
> Key: HBASE-21326
> URL: https://issues.apache.org/jira/browse/HBASE-21326
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Major
>
> I get a few of these each time I do a startup.
> {code}
> 2018-10-16 14:06:42,041 WARN
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker terminating
> UNNATURALLY null
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hbase.procedure2.store.BitSetNode.updateState(BitSetNode.java:396)
> at
> org.apache.hadoop.hbase.procedure2.store.BitSetNode.delete(BitSetNode.java:155)
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:153)
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:138)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.updateStoreTracker(WALProcedureStore.java:782)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:729)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.delete(WALProcedureStore.java:616)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1684)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1475)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2059)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20973) ArrayIndexOutOfBoundsException when rolling back procedure
[
https://issues.apache.org/jira/browse/HBASE-20973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652954#comment-16652954
]
stack commented on HBASE-20973:
---
Here is another example:
{code}
2018-10-16 14:06:47,975 WARN
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Rollback because parent
is done/rolledback proc=pid=1337789, ppid=1275219,
state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
table=IntegrationTestBigLinkedList_20180626064758,
region=cda7c63e2cfee082e8d0d7ee5fc28a20
2018-10-16 14:06:47,976 WARN
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker terminating
UNNATURALLY null
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.hbase.procedure2.store.BitSetNode.updateState(BitSetNode.java:396)
at
org.apache.hadoop.hbase.procedure2.store.BitSetNode.delete(BitSetNode.java:155)
at
org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:153)
at
org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:138)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.updateStoreTracker(WALProcedureStore.java:782)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:729)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.delete(WALProcedureStore.java:616)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1684)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1475)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2059)
{code}
See it on startup.
> ArrayIndexOutOfBoundsException when rolling back procedure
> --
>
> Key: HBASE-20973
> URL: https://issues.apache.org/jira/browse/HBASE-20973
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
>
> Find this one while investigating HBASE-20921. After the root
> procedure(ModifyTableProcedure in this case) rolled back, a
> ArrayIndexOutOfBoundsException was thrown
> {code}
> 2018-07-18 01:39:10,241 ERROR [PEWorker-8] procedure2.ProcedureExecutor(159):
> CODE-BUG: Uncaught runtime exception for pid=5973,
> state=FAILED:MODIFY_TABLE_REOPEN_ALL_REGIONS, exception=java.lang.NullPo
> interException via CODE-BUG: Uncaught runtime exception: pid=5974, ppid=5973,
> state=RUNNABLE:REOPEN_TABLE_REGIONS_CONFIRM_REOPENED;
> ReopenTableRegionsProcedure table=IntegrationTestBigLinkedList:java.l
> ang.NullPointerException; ModifyTableProcedure
> table=IntegrationTestBigLinkedList
> java.lang.UnsupportedOperationException: unhandled
> state=MODIFY_TABLE_REOPEN_ALL_REGIONS
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.rollbackState(ModifyTableProcedure.java:147)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.rollbackState(ModifyTableProcedure.java:50)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:203)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:864)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1353)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1309)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1178)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1741)
> 2018-07-18 01:39:10,243 WARN [PEWorker-8]
> procedure2.ProcedureExecutor(1756): Worker terminating UNNATURALLY null
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker$BitSetNode.updateState(ProcedureStoreTracker.java:405)
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker$BitSetNode.delete(ProcedureStoreTracker.java:178)
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:513)
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:505)
> at
>
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652941#comment-16652941
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #20 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/20/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/20//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/20//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/20//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21326) [amv2] Worker terminating UNNATURALLY null java.lang.ArrayIndexOutOfBoundsException: 1
[
https://issues.apache.org/jira/browse/HBASE-21326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652931#comment-16652931
]
Allan Yang commented on HBASE-21326:
It is duplicated by HBASE-20973, but I haven't figure it out yet.
> [amv2] Worker terminating UNNATURALLY null
> java.lang.ArrayIndexOutOfBoundsException: 1
> --
>
> Key: HBASE-21326
> URL: https://issues.apache.org/jira/browse/HBASE-21326
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Major
>
> I get a few of these each time I do a startup.
> {code}
> 2018-10-16 14:06:42,041 WARN
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker terminating
> UNNATURALLY null
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hbase.procedure2.store.BitSetNode.updateState(BitSetNode.java:396)
> at
> org.apache.hadoop.hbase.procedure2.store.BitSetNode.delete(BitSetNode.java:155)
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:153)
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:138)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.updateStoreTracker(WALProcedureStore.java:782)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:729)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.delete(WALProcedureStore.java:616)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1684)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1475)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2059)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21327) Fix minor logging issue where we don't report servername if no associated SCP
[
https://issues.apache.org/jira/browse/HBASE-21327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21327:
--
Fix Version/s: 2.1.1
2.2.0
3.0.0
Status: Patch Available (was: Open)
Minor.
Fix the above problem and fix fact that we log Start of Stochastic Load
Balancer run but not the end and then make the taking of xlock log more
readable. Minor stuff.
> Fix minor logging issue where we don't report servername if no associated SCP
> -
>
> Key: HBASE-21327
> URL: https://issues.apache.org/jira/browse/HBASE-21327
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Trivial
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21327.branch-2.1.001.patch
>
>
> When reporting on whether an associated SCP, this is what we log:
> {code}
> 2018-10-16 14:05:57,607 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,607 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,607 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,607 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: true has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> {code}
> i.e. we are supposed to log servername but we don't. Minor issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21327) Fix minor logging issue where we don't report servername if no associated SCP
[
https://issues.apache.org/jira/browse/HBASE-21327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21327:
--
Attachment: HBASE-21327.branch-2.1.001.patch
> Fix minor logging issue where we don't report servername if no associated SCP
> -
>
> Key: HBASE-21327
> URL: https://issues.apache.org/jira/browse/HBASE-21327
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Trivial
> Attachments: HBASE-21327.branch-2.1.001.patch
>
>
> When reporting on whether an associated SCP, this is what we log:
> {code}
> 2018-10-16 14:05:57,607 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,607 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,607 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,607 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: true has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> 2018-10-16 14:05:57,608 ERROR
> org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
> ServerCrashProcedure
> {code}
> i.e. we are supposed to log servername but we don't. Minor issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21327) Fix minor logging issue where we don't report servername if no associated SCP
stack created HBASE-21327:
-
Summary: Fix minor logging issue where we don't report servername
if no associated SCP
Key: HBASE-21327
URL: https://issues.apache.org/jira/browse/HBASE-21327
Project: HBase
Issue Type: Bug
Components: amv2
Reporter: stack
Assignee: stack
When reporting on whether an associated SCP, this is what we log:
{code}
2018-10-16 14:05:57,607 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,607 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,607 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,607 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: true has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
2018-10-16 14:05:57,608 ERROR
org.apache.hadoop.hbase.master.RegionServerTracker: false has no matching
ServerCrashProcedure
{code}
i.e. we are supposed to log servername but we don't. Minor issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652899#comment-16652899
]
Ted Yu edited comment on HBASE-21246 at 10/17/18 4:16 AM:
--
bq. the strange thing is that we have a method which creates WALIdentity
directly from a String.
Good point.
In ReplicationSourceManager, we can change:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
to:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
This way, in refreshSources(), we retrieve WALIdentity from the Map and don't
need to create identity.
I am making changes in ReplicationSourceManager and SyncReplicationWALProvider
in wal-refactor repo for evaluation.
I will drop createWALIdentity in the next patch once evaluation passes.
was (Author: yuzhih...@gmail.com):
bq. the strange thing is that we have a method which creates WALIdentity
directly from a String.
Good point.
In ReplicationSourceManager, we can change:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
to:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
This way, in refreshSources(), we retrieve WALIdentity from the Map and don't
need to create identity.
I will drop createWALIdentity in the next patch.
> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: HBASE-20952
>
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch,
> 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch,
> 21246.HBASE-20952.005.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent
> filename in distributed filesystem environment or, the name of the stream
> when the WAL is backed by log stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21326) [amv2] Worker terminating UNNATURALLY null java.lang.ArrayIndexOutOfBoundsException: 1
[
https://issues.apache.org/jira/browse/HBASE-21326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652911#comment-16652911
]
stack commented on HBASE-21326:
---
Here is another example:
{code}
2018-10-16 14:06:47,975 WARN
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Rollback because parent
is done/rolledback proc=pid=1337789, ppid=1275219,
state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
table=IntegrationTestBigLinkedList_20180626064758,
region=cda7c63e2cfee082e8d0d7ee5fc28a20
2018-10-16 14:06:47,976 WARN
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker terminating
UNNATURALLY null
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.hbase.procedure2.store.BitSetNode.updateState(BitSetNode.java:396)
at
org.apache.hadoop.hbase.procedure2.store.BitSetNode.delete(BitSetNode.java:155)
at
org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:153)
at
org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:138)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.updateStoreTracker(WALProcedureStore.java:782)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:729)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.delete(WALProcedureStore.java:616)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1684)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1475)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2059)
{code}
> [amv2] Worker terminating UNNATURALLY null
> java.lang.ArrayIndexOutOfBoundsException: 1
> --
>
> Key: HBASE-21326
> URL: https://issues.apache.org/jira/browse/HBASE-21326
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Major
>
> I get a few of these each time I do a startup.
> {code}
> 2018-10-16 14:06:42,041 WARN
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker terminating
> UNNATURALLY null
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hbase.procedure2.store.BitSetNode.updateState(BitSetNode.java:396)
> at
> org.apache.hadoop.hbase.procedure2.store.BitSetNode.delete(BitSetNode.java:155)
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:153)
> at
> org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:138)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.updateStoreTracker(WALProcedureStore.java:782)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:729)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.delete(WALProcedureStore.java:616)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1684)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1475)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2059)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21326) [amv2] Worker terminating UNNATURALLY null java.lang.ArrayIndexOutOfBoundsException: 1
stack created HBASE-21326:
-
Summary: [amv2] Worker terminating UNNATURALLY null
java.lang.ArrayIndexOutOfBoundsException: 1
Key: HBASE-21326
URL: https://issues.apache.org/jira/browse/HBASE-21326
Project: HBase
Issue Type: Bug
Reporter: stack
I get a few of these each time I do a startup.
{code}
2018-10-16 14:06:42,041 WARN
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Worker terminating
UNNATURALLY null
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.hbase.procedure2.store.BitSetNode.updateState(BitSetNode.java:396)
at
org.apache.hadoop.hbase.procedure2.store.BitSetNode.delete(BitSetNode.java:155)
at
org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:153)
at
org.apache.hadoop.hbase.procedure2.store.ProcedureStoreTracker.delete(ProcedureStoreTracker.java:138)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.updateStoreTracker(WALProcedureStore.java:782)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:729)
at
org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.delete(WALProcedureStore.java:616)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1684)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1475)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2059)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652904#comment-16652904 ] stack commented on HBASE-21322: --- [~tianjingyun] bq. ... if one RS is down, then all procedure logs are deleted, it will lead to that no ServerCrashProcedure is scheduled. Can you say more about the above condition. I do not follow. Thanks. > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21246) Introduce WALIdentity interface
[
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652899#comment-16652899
]
Ted Yu commented on HBASE-21246:
bq. the strange thing is that we have a method which creates WALIdentity
directly from a String.
Good point.
In ReplicationSourceManager, we can change:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
to:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
This way, in refreshSources(), we retrieve WALIdentity from the Map and don't
need to create identity.
I will drop createWALIdentity in the next patch.
> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: HBASE-20952
>
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch,
> 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch,
> 21246.HBASE-20952.005.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent
> filename in distributed filesystem environment or, the name of the stream
> when the WAL is backed by log stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21269) Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state in RegionStates when assign/unassign"
[ https://issues.apache.org/jira/browse/HBASE-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21269: - Attachment: HBASE-21269.master.003.patch > Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state > in RegionStates when assign/unassign" > --- > > Key: HBASE-21269 > URL: https://issues.apache.org/jira/browse/HBASE-21269 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21269.branch-2.001.patch, > HBASE-21269.master.001.patch, HBASE-21269.master.003.patch > > > A bunch of this patch does not apply to branch-2 and master now we don't have > AP or UP anymore. Need to figure if we need override in branch-2 and master. > Let me upload the forward-port done so far. Can finish this when move to > branch-2.2 exercise. FYI [~Apache9] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21269) Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state in RegionStates when assign/unassign"
[ https://issues.apache.org/jira/browse/HBASE-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21269: - Attachment: (was: HBASE-21269.master.002.patch) > Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state > in RegionStates when assign/unassign" > --- > > Key: HBASE-21269 > URL: https://issues.apache.org/jira/browse/HBASE-21269 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21269.branch-2.001.patch, > HBASE-21269.master.001.patch > > > A bunch of this patch does not apply to branch-2 and master now we don't have > AP or UP anymore. Need to figure if we need override in branch-2 and master. > Let me upload the forward-port done so far. Can finish this when move to > branch-2.2 exercise. FYI [~Apache9] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21269) Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state in RegionStates when assign/unassign"
[
https://issues.apache.org/jira/browse/HBASE-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652887#comment-16652887
]
Hadoop QA commented on HBASE-21269:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HBASE-21269 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-21269 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944263/HBASE-21269.master.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14723/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state
> in RegionStates when assign/unassign"
> ---
>
> Key: HBASE-21269
> URL: https://issues.apache.org/jira/browse/HBASE-21269
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21269.branch-2.001.patch,
> HBASE-21269.master.001.patch, HBASE-21269.master.002.patch
>
>
> A bunch of this patch does not apply to branch-2 and master now we don't have
> AP or UP anymore. Need to figure if we need override in branch-2 and master.
> Let me upload the forward-port done so far. Can finish this when move to
> branch-2.2 exercise. FYI [~Apache9]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652883#comment-16652883 ] Jingyun Tian commented on HBASE-21322: -- [~stack] How do you think of this? > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Description: According to my test, if one RS is down, then all procedure logs are deleted, it will lead to that no ServerCrashProcedure is scheduled. And restarting master cannot help. Thus we need to schedule a ServerCrashProcedure manually to solve the problem. I plan to add a scheduleServerCrashProcedure() API to HbckService, then add this API to HBCK2. (was: According to my test, if one RS is down, then all procedure logs are deleted, it will lead to that no ServerCrashProcedure is scheduled. And restarting master cannot help. Thus we need to schedule a ServerCrashProcedure manually to solve the problem. I plan to add a ) > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a scheduleServerCrashProcedure() API to > HbckService, then add this API to HBCK2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Description: According to my test, if one RS is down, then all procedure logs are deleted, it will lead to that no ServerCrashProcedure is scheduled. And restarting master cannot help. Thus we need to schedule a ServerCrashProcedure manually to solve the problem. I plan to add a (was: According to my test, if one RS is down, then all procedure logs are deleted, it will lead to that no ServerCrashProcedure is scheduled. And restarting master cannot help. Thus we need to schedule a ServerCrashProcedure manually to solve the problem.) > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. I plan to add a -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[ https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21322: - Issue Type: Sub-task (was: Task) Parent: HBASE-19121 > Add a scheduleServerCrashProcedure() API to HbckService > --- > > Key: HBASE-21322 > URL: https://issues.apache.org/jira/browse/HBASE-21322 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > > According to my test, if one RS is down, then all procedure logs are deleted, > it will lead to that no ServerCrashProcedure is scheduled. And restarting > master cannot help. Thus we need to schedule a ServerCrashProcedure manually > to solve the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21269) Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state in RegionStates when assign/unassign"
[ https://issues.apache.org/jira/browse/HBASE-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21269: - Attachment: HBASE-21269.master.002.patch > Forward-port to branch-2 " HBASE-21213 [hbck2] bypass leaves behind state > in RegionStates when assign/unassign" > --- > > Key: HBASE-21269 > URL: https://issues.apache.org/jira/browse/HBASE-21269 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21269.branch-2.001.patch, > HBASE-21269.master.001.patch, HBASE-21269.master.002.patch > > > A bunch of this patch does not apply to branch-2 and master now we don't have > AP or UP anymore. Need to figure if we need override in branch-2 and master. > Let me upload the forward-port done so far. Can finish this when move to > branch-2.2 exercise. FYI [~Apache9] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21200) Memstore flush doesn't finish because of seekToPreviousRow() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-21200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652853#comment-16652853 ] Toshihiro Suzuki commented on HBASE-21200: -- Thank you for reviewing. [~ram_krish] Yes, we should have one more reviewer as the patch modifies critical part. [~anoop.hbase] [~yuzhih...@gmail.com] [~apurtell] Could you please review v2 patch if you have time? > Memstore flush doesn't finish because of seekToPreviousRow() in memstore > scanner. > - > > Key: HBASE-21200 > URL: https://issues.apache.org/jira/browse/HBASE-21200 > Project: HBase > Issue Type: Bug > Components: Scanners >Reporter: dongjin2193.jeon >Assignee: Toshihiro Suzuki >Priority: Critical > Attachments: HBASE-21200-UT.patch, HBASE-21200.master.001.patch, > HBASE-21200.master.002.patch, RegionServerJstack.log > > > The issue of delaying memstore flush still occurs after backport hbase-15871. > Reverse scan takes a long time to seek previous row in the memstore full of > deleted cells. > > jstack : > "MemStoreFlusher.0" #114 prio=5 os_prio=0 tid=0x7fa3d0729000 nid=0x486a > waiting on condition [0x7fa3b9b6b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xa465fe60> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) > at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > at > org.apache.hadoop.hbase.regionserver.*StoreScanner.updateReaders(StoreScanner.java:695)* > at > org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1127) > at > org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1106) > at > org.apache.hadoop.hbase.regionserver.HStore.access$600(HStore.java:130) > at > org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2455) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2519) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2256) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2218) > at > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2110) > at > org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2036) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:501) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) > at java.lang.Thread.run(Thread.java:748) > > "RpcServer.FifoWFPBQ.default.handler=27,queue=0,port=16020" #65 daemon prio=5 > os_prio=0 tid=0x7fa3e628 nid=0x4801 runnable [0x7fa3bd29a000] > java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.getNext(DefaultMemStore.java:780) > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekInSubLists(DefaultMemStore.java:826) > - locked <0xb45aa5b8> (a > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner) > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seek(DefaultMemStore.java:818) > - locked <0xb45aa5b8> (a > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner) > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekToPreviousRow(DefaultMemStore.java:1000) > - locked <0xb45aa5b8> (a > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner) > at > org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:136) > at > org.apache.hadoop.hbase.regionserver.*StoreScanner.next(StoreScanner.java:629)* > at >
[jira] [Commented] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652832#comment-16652832 ] Duo Zhang commented on HBASE-21246: --- For me I'm OK with the WALIdentity but the strange thing is that we have a method which creates WALIdentity directly from a String. What I thought is that, there is no way to 'create' a WALIdentity, you can only get WALIdentities from the WAL system. And for the preWALRoll, I think we need revisit the CP APIs. Maybe there will be no log rolling for some WAL implementation. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch, > 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch, > 21246.HBASE-20952.005.patch > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21320) [canary] Cleanup of usage and add commentary
[
https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652820#comment-16652820
]
Hadoop QA commented on HBASE-21320:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
54s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
28s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
48s{color} | {color:green} branch-2.1 passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
32s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
45s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
22s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
23s{color} | {color:red} hbase-server: The patch generated 2 new + 24 unchanged
- 33 fixed = 26 total (was 57) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
23s{color} | {color:red} root: The patch generated 2 new + 34 unchanged - 33
fixed = 36 total (was 67) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
44s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 39s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}212m 56s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} |
[jira] [Created] (HBASE-21325) Add a max wait time for waitOnAllRegionsToClose
Duo Zhang created HBASE-21325: - Summary: Add a max wait time for waitOnAllRegionsToClose Key: HBASE-21325 URL: https://issues.apache.org/jira/browse/HBASE-21325 Project: HBase Issue Type: Improvement Reporter: Duo Zhang When testing sync replication, I found that, if I transit the remote cluster to DA, while the local cluster is still in A, the region server will hang when shutdown. As the fsOk flag only test the local cluster(which is reasonable), we will enter the waitOnAllRegionsToClose, and since the WAL is broken(the remote wal directory is gone) so we will never succeed. And this lead to an infinite wait inside waitOnAllRegionsToClose. So I think here we should have an upper bound for the wait time in waitOnAllRegionsToClose method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21319) The nightly jobs should run 'mvn test' with '-fn'
[ https://issues.apache.org/jira/browse/HBASE-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652818#comment-16652818 ] Duo Zhang commented on HBASE-21319: --- And there are lots of blank on the flakey dashboard https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/master/lastSuccessfulBuild/artifact/dashboard.html Maybe the problem is that we do not run these tests in nightly, or there are parsing errors? > The nightly jobs should run 'mvn test' with '-fn' > - > > Key: HBASE-21319 > URL: https://issues.apache.org/jira/browse/HBASE-21319 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Duo Zhang >Priority: Major > > For now, it will fail in the middle if there are failures. I think for a > nightly job we should run all the UTs. > The nightly pipeline is complicated and uses yetus so I'm not sure how to fix > it... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21319) The nightly jobs should run 'mvn test' with '-fn'
[ https://issues.apache.org/jira/browse/HBASE-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652816#comment-16652816 ] Duo Zhang commented on HBASE-21319: --- I mean the nightly job, not the pre commit job... In nightly job we will not pull out the flakey tests? > The nightly jobs should run 'mvn test' with '-fn' > - > > Key: HBASE-21319 > URL: https://issues.apache.org/jira/browse/HBASE-21319 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Duo Zhang >Priority: Major > > For now, it will fail in the middle if there are failures. I think for a > nightly job we should run all the UTs. > The nightly pipeline is complicated and uses yetus so I'm not sure how to fix > it... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21323) The holdingCleanupTracker is not updated properly and causes too many procedure wal files
[
https://issues.apache.org/jira/browse/HBASE-21323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652809#comment-16652809
]
Duo Zhang commented on HBASE-21323:
---
Oh I think the problem is that we will only delete the sub procedures when root
procedure is done. Let me write a UT first and see how to fix.
> The holdingCleanupTracker is not updated properly and causes too many
> procedure wal files
> -
>
> Key: HBASE-21323
> URL: https://issues.apache.org/jira/browse/HBASE-21323
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Priority: Major
>
> Keep seeing this
> {noformat}
> 2018-10-16,20:03:02,027 WARN [WALProcedureStoreSyncThread]
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure
> WALs count=340 above the warning threshold 10. check running procedures to
> see if something is stuck.
> 2018-10-16,20:03:02,027 INFO [WALProcedureStoreSyncThread]
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new
> Procedure Store WAL, id=343
> 2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=991,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=992,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=994,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=995,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:02,870 WARN [WALProcedureStoreSyncThread]
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure
> WALs count=341 above the warning threshold 10. check running procedures to
> see if something is stuck.
> 2018-10-16,20:03:02,870 INFO [WALProcedureStoreSyncThread]
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new
> Procedure Store WAL, id=344
> 2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=991,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=992,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=994,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=995,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:03,816 WARN [WALProcedureStoreSyncThread]
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure
> WALs count=342 above the warning threshold 10. check running procedures to
> see if something is stuck.
> 2018-10-16,20:03:03,816 INFO [WALProcedureStoreSyncThread]
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new
> Procedure Store WAL, id=345
> 2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=991,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=992,
> ppid=990, state=SUCCESS;
> org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already
> been finished, skip force updating.
> 2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0]
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor:
[jira] [Commented] (HBASE-21224) Handle compaction queue duplication
[
https://issues.apache.org/jira/browse/HBASE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652799#comment-16652799
]
Allan Yang commented on HBASE-21224:
{quote}
There is no efficient way ( O(1) ) to remove an element from
ThreadPoolExecutor.
{quote}
It is fine to iterate over the queue to delete the element inside. Just be
simple, no data structure change needed.
> Handle compaction queue duplication
> ---
>
> Key: HBASE-21224
> URL: https://issues.apache.org/jira/browse/HBASE-21224
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
>
> Mentioned by [~allan163] that we may want to handle compaction queue
> duplication in this Jira https://issues.apache.org/jira/browse/HBASE-18451
> Creating this item for further assessment and discussion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21284) Forward port HBASE-21000 to branch-2
[ https://issues.apache.org/jira/browse/HBASE-21284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reassigned HBASE-21284: - Assignee: Mingliang Liu > Forward port HBASE-21000 to branch-2 > > > Key: HBASE-21284 > URL: https://issues.apache.org/jira/browse/HBASE-21284 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Mingliang Liu >Priority: Major > > See parent for details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[
https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652764#comment-16652764
]
Hadoop QA commented on HBASE-21073:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
4s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
23s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
20s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 1s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
44s{color} | {color:green} hbase-zookeeper in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}234m 26s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}286m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure |
| | hadoop.hbase.master.assignment.TestSplitTableRegionProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21073 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944205/HBASE-21073.master.008.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 50a632059b93 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
[jira] [Commented] (HBASE-21224) Handle compaction queue duplication
[
https://issues.apache.org/jira/browse/HBASE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652760#comment-16652760
]
Xu Cang commented on HBASE-21224:
-
We maintain short compactions in ThreadPoolExecutor. There is no efficient way
( O(1) ) to remove an element from ThreadPoolExecutor.
Also, to inspect the compactions in threadPoolExecutor we need to use
[{{getQueue()}}|https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html#getQueue()]
(allows access to the work queue for purposes of monitoring and debugging. Use
of this method for any other purpose is strongly discouraged) which is not a
good way to do it according to the doc.
If we really need this improvement, may need to think about another type of
data structures.
> Handle compaction queue duplication
> ---
>
> Key: HBASE-21224
> URL: https://issues.apache.org/jira/browse/HBASE-21224
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
>
> Mentioned by [~allan163] that we may want to handle compaction queue
> duplication in this Jira https://issues.apache.org/jira/browse/HBASE-18451
> Creating this item for further assessment and discussion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652720#comment-16652720
]
stack commented on HBASE-21291:
---
Let me test and see if we should pull it back further, into 2.0 and 2.1.
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch, HBASE-21291.master.005.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21320) [canary] Cleanup of usage and add commentary
[
https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652715#comment-16652715
]
Hadoop QA commented on HBASE-21320:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
33s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
36s{color} | {color:green} branch-2.1 passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
20s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
46s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
12s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
11s{color} | {color:red} hbase-server: The patch generated 2 new + 24 unchanged
- 33 fixed = 26 total (was 57) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
2s{color} | {color:red} root: The patch generated 2 new + 34 unchanged - 33
fixed = 36 total (was 67) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
50s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 50s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}178m 34s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
4s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} |
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652661#comment-16652661
]
Hudson commented on HBASE-21291:
Results for branch branch-2
[build #1399 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1399/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1399//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1399//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1399//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch, HBASE-21291.master.005.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652544#comment-16652544
]
Hudson commented on HBASE-21242:
Results for branch branch-2.0
[build #960 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/960/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/960//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/960//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/960//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments:
> 0001-HBASE-21317-hbck2-Add-version-version-handling-and.patch,
> HBASE-21242.addendum.fix.testhregioninfo.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.002.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.002.patch,
> HBASE-21242.branch-2.002.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21263) Mention compression algorithm along with other storefile details
[ https://issues.apache.org/jira/browse/HBASE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652486#comment-16652486 ] Hudson commented on HBASE-21263: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1172 (See [https://builds.apache.org/job/HBase-1.2-IT/1172/]) HBASE-21263 Mention compression algorithm along with other storefile (apurtell: rev b50019d0b3c8c906b5e2642fc4c330ff11a1f71d) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java > Mention compression algorithm along with other storefile details > > > Key: HBASE-21263 > URL: https://issues.apache.org/jira/browse/HBASE-21263 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Subrat Mishra >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9 > > Attachments: HBASE-21263-branch-1.patch, > HBASE-21263-branch-2.0.patch, HBASE-21263.master.001.patch, > HBASE-21263.master.002.patch, HBASE-21263.master.003.patch, HBASE-21263.patch > > > Where we log storefile details we should also log the compression algorithm > used to compress blocks on disk, if any. > For example, here's a log line out of compaction: > 2018-10-02 21:59:47,594 DEBUG > [regionserver/host/1.1.1.1:8120-longCompactions-1538517461152] > compactions.Compactor: Compacting > hdfs://namenode:8020/hbase/data/default/TestTable/86037c19117a46b5b8148439ea55753b/i/3d04a7c28d6343ceb773737dbb192533, > keycount=3335873, bloomtype=ROW, size=107.5 M, encoding=ROW_INDEX_V1, > seqNum=154199, earliestPutTs=1538516084915 > Aside from bloom type, block encoding, and filename, it would be good to know > compression type in this type of DEBUG or INFO level logging. A minor > omission of information that could be helpful during debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21263) Mention compression algorithm along with other storefile details
[ https://issues.apache.org/jira/browse/HBASE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652465#comment-16652465 ] Hudson commented on HBASE-21263: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #492 (See [https://builds.apache.org/job/HBase-1.3-IT/492/]) HBASE-21263 Mention compression algorithm along with other storefile (apurtell: rev 26ef2a016032bc05c2ce6fd1b1a96034413c1ed5) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java > Mention compression algorithm along with other storefile details > > > Key: HBASE-21263 > URL: https://issues.apache.org/jira/browse/HBASE-21263 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Subrat Mishra >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9 > > Attachments: HBASE-21263-branch-1.patch, > HBASE-21263-branch-2.0.patch, HBASE-21263.master.001.patch, > HBASE-21263.master.002.patch, HBASE-21263.master.003.patch, HBASE-21263.patch > > > Where we log storefile details we should also log the compression algorithm > used to compress blocks on disk, if any. > For example, here's a log line out of compaction: > 2018-10-02 21:59:47,594 DEBUG > [regionserver/host/1.1.1.1:8120-longCompactions-1538517461152] > compactions.Compactor: Compacting > hdfs://namenode:8020/hbase/data/default/TestTable/86037c19117a46b5b8148439ea55753b/i/3d04a7c28d6343ceb773737dbb192533, > keycount=3335873, bloomtype=ROW, size=107.5 M, encoding=ROW_INDEX_V1, > seqNum=154199, earliestPutTs=1538516084915 > Aside from bloom type, block encoding, and filename, it would be good to know > compression type in this type of DEBUG or INFO level logging. A minor > omission of information that could be helpful during debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21263) Mention compression algorithm along with other storefile details
[ https://issues.apache.org/jira/browse/HBASE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21263: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.9 2.0.3 2.1.1 2.2.0 1.2.8 1.3.3 1.5.0 3.0.0 Status: Resolved (was: Patch Available) Pushed everywhere. Minor changes needed on branch-2.0 and the branch-1s. I also removed one instance of logging in memstore that didn't seem appropriate, and I didn't think we needed a round of bikeshedding for that... > Mention compression algorithm along with other storefile details > > > Key: HBASE-21263 > URL: https://issues.apache.org/jira/browse/HBASE-21263 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Subrat Mishra >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9 > > Attachments: HBASE-21263-branch-1.patch, > HBASE-21263-branch-2.0.patch, HBASE-21263.master.001.patch, > HBASE-21263.master.002.patch, HBASE-21263.master.003.patch, HBASE-21263.patch > > > Where we log storefile details we should also log the compression algorithm > used to compress blocks on disk, if any. > For example, here's a log line out of compaction: > 2018-10-02 21:59:47,594 DEBUG > [regionserver/host/1.1.1.1:8120-longCompactions-1538517461152] > compactions.Compactor: Compacting > hdfs://namenode:8020/hbase/data/default/TestTable/86037c19117a46b5b8148439ea55753b/i/3d04a7c28d6343ceb773737dbb192533, > keycount=3335873, bloomtype=ROW, size=107.5 M, encoding=ROW_INDEX_V1, > seqNum=154199, earliestPutTs=1538516084915 > Aside from bloom type, block encoding, and filename, it would be good to know > compression type in this type of DEBUG or INFO level logging. A minor > omission of information that could be helpful during debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21263) Mention compression algorithm along with other storefile details
[ https://issues.apache.org/jira/browse/HBASE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21263: --- Attachment: HBASE-21263.patch HBASE-21263-branch-2.0.patch HBASE-21263-branch-1.patch > Mention compression algorithm along with other storefile details > > > Key: HBASE-21263 > URL: https://issues.apache.org/jira/browse/HBASE-21263 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Subrat Mishra >Priority: Minor > Labels: beginner, beginners > Attachments: HBASE-21263-branch-1.patch, > HBASE-21263-branch-2.0.patch, HBASE-21263.master.001.patch, > HBASE-21263.master.002.patch, HBASE-21263.master.003.patch, HBASE-21263.patch > > > Where we log storefile details we should also log the compression algorithm > used to compress blocks on disk, if any. > For example, here's a log line out of compaction: > 2018-10-02 21:59:47,594 DEBUG > [regionserver/host/1.1.1.1:8120-longCompactions-1538517461152] > compactions.Compactor: Compacting > hdfs://namenode:8020/hbase/data/default/TestTable/86037c19117a46b5b8148439ea55753b/i/3d04a7c28d6343ceb773737dbb192533, > keycount=3335873, bloomtype=ROW, size=107.5 M, encoding=ROW_INDEX_V1, > seqNum=154199, earliestPutTs=1538516084915 > Aside from bloom type, block encoding, and filename, it would be good to know > compression type in this type of DEBUG or INFO level logging. A minor > omission of information that could be helpful during debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[
https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652432#comment-16652432
]
Hadoop QA commented on HBASE-21073:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
16s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
25s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 52s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hbase-zookeeper in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}139m
58s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}186m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21073 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944184/HBASE-21073.master.007.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 090a91d9f5e5 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 821e4d7de2 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
|
[jira] [Updated] (HBASE-21216) TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
[
https://issues.apache.org/jira/browse/HBASE-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21216:
---
Fix Version/s: 3.0.0
> TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
> --
>
> Key: HBASE-21216
> URL: https://issues.apache.org/jira/browse/HBASE-21216
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21216.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
> {code}
> java.lang.AssertionError: Archived hfiles [] and table hfiles
> [9ca09392705f425f9c916beedc10d63c] is missing snapshot
> file:6739a09747e54189a4112a6d8f37e894
> at
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
> {code}
> The file appeared in archive dir before hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |data/
> 2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |---default/
> 2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |--test/
> 2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-1237d57b63a7bdf067a930441a02514a/
> 2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |recovered.edits/
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---4.seqid
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-29e1700e09b51223ad2f5811105a4d51/
> 2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |fam/
> 2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---2c66a18f6c1a4074b84ffbb3245268c4
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---45bb396c6a5e49629e45a4d56f1e9b14
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---6739a09747e54189a4112a6d8f37e894
> {code}
> However, the archive dir became empty after hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-corrupt/
> {code}
> Leading to the assertion failure.
> This test is one of the top flaky tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21279) Split TestAdminShell into several tests
[
https://issues.apache.org/jira/browse/HBASE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652408#comment-16652408
]
Hadoop QA commented on HBASE-21279:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
7s{color} | {color:red} The patch generated 3 new + 123 unchanged - 179 fixed =
126 total (was 302) {color} |
| {color:orange}-0{color} | {color:orange} ruby-lint {color} | {color:orange}
0m 16s{color} | {color:orange} The patch generated 1 new + 274 unchanged - 328
fixed = 275 total (was 602) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 27s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
44s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21279 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944201/HBASE-21279.v01.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile rubocop ruby_lint |
| uname | Linux a8d5b65292f8 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 821e4d7de2 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| rubocop | v0.59.2 |
| rubocop |
[jira] [Commented] (HBASE-21285) Enhanced TableSnapshotInputFormat to allow a size based splitting
[
https://issues.apache.org/jira/browse/HBASE-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652390#comment-16652390
]
Hadoop QA commented on HBASE-21285:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
2s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} hbase-mapreduce: The patch generated 2 new + 15
unchanged - 0 fixed = 17 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 44s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 27m
57s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21285 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944197/HBASE-21285.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1f16bedd78b6 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 821e4d7de2 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14718/artifact/patchprocess/diff-checkstyle-hbase-mapreduce.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14718/testReport/ |
| Max. process+thread count | 5203 (vs. ulimit of 1) |
| modules | C: hbase-mapreduce U: hbase-mapreduce |
| Console
[jira] [Comment Edited] (HBASE-21279) Split TestAdminShell into several tests
[
https://issues.apache.org/jira/browse/HBASE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652387#comment-16652387
]
Ted Yu edited comment on HBASE-21279 at 10/16/18 8:21 PM:
--
Thanks for working on this, Artem.
{code}
+module Hbase
+ class AdminHelpersTest < Test::Unit::TestCase
+include TestHelpers
{code}
I was expecting the new test class to have different name from the existing one.
{code}
System.setProperty("shell.test.include", "admin_test.rb");
+System.setProperty("shell.test.include", "admin_test2.rb");
{code}
Isn't the net effect of the above "admin_test2.rb" being the sole test being
run ?
was (Author: yuzhih...@gmail.com):
{code}
+module Hbase
+ class AdminHelpersTest < Test::Unit::TestCase
+include TestHelpers
{code}
I was expecting the new test class to have different name from the existing one.
{code}
System.setProperty("shell.test.include", "admin_test.rb");
+System.setProperty("shell.test.include", "admin_test2.rb");
{code}
Isn't the net effect of the above "admin_test2.rb" being the sole test being
run ?
> Split TestAdminShell into several tests
> ---
>
> Key: HBASE-21279
> URL: https://issues.apache.org/jira/browse/HBASE-21279
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-21279.v01.patch, testAdminShell-output.tar.gz
>
>
> In the flaky test board, TestAdminShell often timed out
> (https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html).
> I ran the test on Linux with SSD and reproduced the timeout (see attached
> test output).
> {code}
> 2018-10-08 02:36:09,146 DEBUG [main] hbase.HBaseTestingUtility(351): Setting
> hbase.rootdir to
> /mnt/disk2/a/2-hbase/hbase-shell/target/test-data/a103d8e4-695c-a5a9-6690-1ef2580050f9
> ...
> 2018-10-08 02:49:09,093 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=7]
> master.MasterRpcServices(1171): Checking to see if procedure is done pid=871
> Took 0.7262 seconds2018-10-08 02:49:09,324 DEBUG [PEWorker-1]
> util.FSTableDescriptors(684): Wrote into
> hdfs://localhost:43859/user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-
> d935316c9241/.tmp/data/default/hbase_shell_tests_table/.tabledesc/.tableinfo.01
> 2018-10-08 02:49:09,328 INFO
> [RegionOpenAndInitThread-hbase_shell_tests_table-1]
> regionserver.HRegion(7004): creating HRegion hbase_shell_tests_table HTD ==
> 'hbase_shell_tests_table', {NAME => 'x', VERSIONS => '5',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE =>
> 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS =>
> '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'}, {NAME => 'y', VERSIONS => '1',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
> DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0',
> REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:43859/
> user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-d935316c9241/.tmp Table name ==
> hbase_shell_tests_table
> ^[[38;5;226mE^[[0m
> ===
> Error: ^[[48;5;16;38;5;226;1mtest_Get_simple_status(Hbase::StatusTest)^[[0m:
> Java::JavaIo::InterruptedIOException: Interrupt while waiting on Operation:
> CREATE, Table Name: default:hbase_shell_tests_table, procId: 871
> 2018-10-08 02:49:09,361 INFO [Block report processor]
> blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated:
> 127.0.0.1:41338 is added to
> blk_1073742193_1369{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-ecc89143-e0a5-4a1c-b552-120be2561334:NORMAL:127.0.0.1:
>41338|RBW]]} size 58
> > TEST TIMED OUT. PRINTING THREAD DUMP. <
> {code}
> We can see that the procedure #871 wasn't stuck - the timeout cut in and
> stopped the test.
> We should separate the current test into two (or more) test files (with
> corresponding .rb) so that the execution time consistently would not exceed
> limit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21279) Split TestAdminShell into several tests
[
https://issues.apache.org/jira/browse/HBASE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652387#comment-16652387
]
Ted Yu commented on HBASE-21279:
{code}
+module Hbase
+ class AdminHelpersTest < Test::Unit::TestCase
+include TestHelpers
{code}
I was expecting the new test class to have different name from the existing one.
{code}
System.setProperty("shell.test.include", "admin_test.rb");
+System.setProperty("shell.test.include", "admin_test2.rb");
{code}
Isn't the net effect of the above "admin_test2.rb" being the sole test being
run ?
> Split TestAdminShell into several tests
> ---
>
> Key: HBASE-21279
> URL: https://issues.apache.org/jira/browse/HBASE-21279
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-21279.v01.patch, testAdminShell-output.tar.gz
>
>
> In the flaky test board, TestAdminShell often timed out
> (https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html).
> I ran the test on Linux with SSD and reproduced the timeout (see attached
> test output).
> {code}
> 2018-10-08 02:36:09,146 DEBUG [main] hbase.HBaseTestingUtility(351): Setting
> hbase.rootdir to
> /mnt/disk2/a/2-hbase/hbase-shell/target/test-data/a103d8e4-695c-a5a9-6690-1ef2580050f9
> ...
> 2018-10-08 02:49:09,093 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=7]
> master.MasterRpcServices(1171): Checking to see if procedure is done pid=871
> Took 0.7262 seconds2018-10-08 02:49:09,324 DEBUG [PEWorker-1]
> util.FSTableDescriptors(684): Wrote into
> hdfs://localhost:43859/user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-
> d935316c9241/.tmp/data/default/hbase_shell_tests_table/.tabledesc/.tableinfo.01
> 2018-10-08 02:49:09,328 INFO
> [RegionOpenAndInitThread-hbase_shell_tests_table-1]
> regionserver.HRegion(7004): creating HRegion hbase_shell_tests_table HTD ==
> 'hbase_shell_tests_table', {NAME => 'x', VERSIONS => '5',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE =>
> 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS =>
> '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'}, {NAME => 'y', VERSIONS => '1',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
> DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0',
> REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:43859/
> user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-d935316c9241/.tmp Table name ==
> hbase_shell_tests_table
> ^[[38;5;226mE^[[0m
> ===
> Error: ^[[48;5;16;38;5;226;1mtest_Get_simple_status(Hbase::StatusTest)^[[0m:
> Java::JavaIo::InterruptedIOException: Interrupt while waiting on Operation:
> CREATE, Table Name: default:hbase_shell_tests_table, procId: 871
> 2018-10-08 02:49:09,361 INFO [Block report processor]
> blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated:
> 127.0.0.1:41338 is added to
> blk_1073742193_1369{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-ecc89143-e0a5-4a1c-b552-120be2561334:NORMAL:127.0.0.1:
>41338|RBW]]} size 58
> > TEST TIMED OUT. PRINTING THREAD DUMP. <
> {code}
> We can see that the procedure #871 wasn't stuck - the timeout cut in and
> stopped the test.
> We should separate the current test into two (or more) test files (with
> corresponding .rb) so that the execution time consistently would not exceed
> limit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21320) [canary] Cleanup of usage and add commentary
[ https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21320: -- Attachment: HBASE-21320.branch-2.1.004.patch > [canary] Cleanup of usage and add commentary > > > Key: HBASE-21320 > URL: https://issues.apache.org/jira/browse/HBASE-21320 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21320.branch-2.1.001.patch, > HBASE-21320.branch-2.1.002.patch, HBASE-21320.branch-2.1.003.patch, > HBASE-21320.branch-2.1.004.patch > > > I was trying to us the Canary as a diagnosis tool figuring what was online > and what was off but I couldn't make sense of how to use it. There was a bug > where we default to regionserver 'mode' always and only way to change it was > via a system property and a cryptic class name. > Changed it so we don't specify output ahead of figuring what 'mode' we are to > run in. Cleaned-up usage and added commentary. It seems useable now (Did not > change command-line args though they need changing). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21320) [canary] Cleanup of usage and add commentary
[ https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652380#comment-16652380 ] stack commented on HBASE-21320: --- .003 addresses mistake in doc found by [~psomogyi] > [canary] Cleanup of usage and add commentary > > > Key: HBASE-21320 > URL: https://issues.apache.org/jira/browse/HBASE-21320 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21320.branch-2.1.001.patch, > HBASE-21320.branch-2.1.002.patch, HBASE-21320.branch-2.1.003.patch > > > I was trying to us the Canary as a diagnosis tool figuring what was online > and what was off but I couldn't make sense of how to use it. There was a bug > where we default to regionserver 'mode' always and only way to change it was > via a system property and a cryptic class name. > Changed it so we don't specify output ahead of figuring what 'mode' we are to > run in. Cleaned-up usage and added commentary. It seems useable now (Did not > change command-line args though they need changing). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21320) [canary] Cleanup of usage and add commentary
[ https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21320: -- Attachment: HBASE-21320.branch-2.1.003.patch > [canary] Cleanup of usage and add commentary > > > Key: HBASE-21320 > URL: https://issues.apache.org/jira/browse/HBASE-21320 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21320.branch-2.1.001.patch, > HBASE-21320.branch-2.1.002.patch, HBASE-21320.branch-2.1.003.patch > > > I was trying to us the Canary as a diagnosis tool figuring what was online > and what was off but I couldn't make sense of how to use it. There was a bug > where we default to regionserver 'mode' always and only way to change it was > via a system property and a cryptic class name. > Changed it so we don't specify output ahead of figuring what 'mode' we are to > run in. Cleaned-up usage and added commentary. It seems useable now (Did not > change command-line args though they need changing). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21324) [balancer] Gives up too soon when lots of regions
stack created HBASE-21324: - Summary: [balancer] Gives up too soon when lots of regions Key: HBASE-21324 URL: https://issues.apache.org/jira/browse/HBASE-21324 Project: HBase Issue Type: Bug Components: Balancer Reporter: stack Assignee: stack When there a few regions, the balancer seems to do a poorish job. My cluster is all out of whack and while the balancer runs, it doesn't do anything. So, I need to configure the balancer to run against large number of regions. There is no doc in reguide to help and there is no logging coming out of the balancer to help either. Let me fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652374#comment-16652374 ] Mike Drob commented on HBASE-21073: --- Added an additional check that repair mode has been activated. > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch, HBASE-21073.master.003.patch, > HBASE-21073.master.004.patch, HBASE-21073.master.005.patch, > HBASE-21073.master.006.patch, HBASE-21073.master.007.patch, > HBASE-21073.master.008.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-21073: -- Attachment: HBASE-21073.master.008.patch > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch, HBASE-21073.master.003.patch, > HBASE-21073.master.004.patch, HBASE-21073.master.005.patch, > HBASE-21073.master.006.patch, HBASE-21073.master.007.patch, > HBASE-21073.master.008.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21279) Split TestAdminShell into several tests
[
https://issues.apache.org/jira/browse/HBASE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Ervits updated HBASE-21279:
-
Status: Patch Available (was: Open)
> Split TestAdminShell into several tests
> ---
>
> Key: HBASE-21279
> URL: https://issues.apache.org/jira/browse/HBASE-21279
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-21279.v01.patch, testAdminShell-output.tar.gz
>
>
> In the flaky test board, TestAdminShell often timed out
> (https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html).
> I ran the test on Linux with SSD and reproduced the timeout (see attached
> test output).
> {code}
> 2018-10-08 02:36:09,146 DEBUG [main] hbase.HBaseTestingUtility(351): Setting
> hbase.rootdir to
> /mnt/disk2/a/2-hbase/hbase-shell/target/test-data/a103d8e4-695c-a5a9-6690-1ef2580050f9
> ...
> 2018-10-08 02:49:09,093 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=7]
> master.MasterRpcServices(1171): Checking to see if procedure is done pid=871
> Took 0.7262 seconds2018-10-08 02:49:09,324 DEBUG [PEWorker-1]
> util.FSTableDescriptors(684): Wrote into
> hdfs://localhost:43859/user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-
> d935316c9241/.tmp/data/default/hbase_shell_tests_table/.tabledesc/.tableinfo.01
> 2018-10-08 02:49:09,328 INFO
> [RegionOpenAndInitThread-hbase_shell_tests_table-1]
> regionserver.HRegion(7004): creating HRegion hbase_shell_tests_table HTD ==
> 'hbase_shell_tests_table', {NAME => 'x', VERSIONS => '5',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE =>
> 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS =>
> '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'}, {NAME => 'y', VERSIONS => '1',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
> DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0',
> REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:43859/
> user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-d935316c9241/.tmp Table name ==
> hbase_shell_tests_table
> ^[[38;5;226mE^[[0m
> ===
> Error: ^[[48;5;16;38;5;226;1mtest_Get_simple_status(Hbase::StatusTest)^[[0m:
> Java::JavaIo::InterruptedIOException: Interrupt while waiting on Operation:
> CREATE, Table Name: default:hbase_shell_tests_table, procId: 871
> 2018-10-08 02:49:09,361 INFO [Block report processor]
> blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated:
> 127.0.0.1:41338 is added to
> blk_1073742193_1369{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-ecc89143-e0a5-4a1c-b552-120be2561334:NORMAL:127.0.0.1:
>41338|RBW]]} size 58
> > TEST TIMED OUT. PRINTING THREAD DUMP. <
> {code}
> We can see that the procedure #871 wasn't stuck - the timeout cut in and
> stopped the test.
> We should separate the current test into two (or more) test files (with
> corresponding .rb) so that the execution time consistently would not exceed
> limit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21279) Split TestAdminShell into several tests
[
https://issues.apache.org/jira/browse/HBASE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Ervits updated HBASE-21279:
-
Attachment: HBASE-21279.v01.patch
> Split TestAdminShell into several tests
> ---
>
> Key: HBASE-21279
> URL: https://issues.apache.org/jira/browse/HBASE-21279
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-21279.v01.patch, testAdminShell-output.tar.gz
>
>
> In the flaky test board, TestAdminShell often timed out
> (https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html).
> I ran the test on Linux with SSD and reproduced the timeout (see attached
> test output).
> {code}
> 2018-10-08 02:36:09,146 DEBUG [main] hbase.HBaseTestingUtility(351): Setting
> hbase.rootdir to
> /mnt/disk2/a/2-hbase/hbase-shell/target/test-data/a103d8e4-695c-a5a9-6690-1ef2580050f9
> ...
> 2018-10-08 02:49:09,093 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=7]
> master.MasterRpcServices(1171): Checking to see if procedure is done pid=871
> Took 0.7262 seconds2018-10-08 02:49:09,324 DEBUG [PEWorker-1]
> util.FSTableDescriptors(684): Wrote into
> hdfs://localhost:43859/user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-
> d935316c9241/.tmp/data/default/hbase_shell_tests_table/.tabledesc/.tableinfo.01
> 2018-10-08 02:49:09,328 INFO
> [RegionOpenAndInitThread-hbase_shell_tests_table-1]
> regionserver.HRegion(7004): creating HRegion hbase_shell_tests_table HTD ==
> 'hbase_shell_tests_table', {NAME => 'x', VERSIONS => '5',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE =>
> 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS =>
> '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'}, {NAME => 'y', VERSIONS => '1',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
> DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0',
> REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:43859/
> user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-d935316c9241/.tmp Table name ==
> hbase_shell_tests_table
> ^[[38;5;226mE^[[0m
> ===
> Error: ^[[48;5;16;38;5;226;1mtest_Get_simple_status(Hbase::StatusTest)^[[0m:
> Java::JavaIo::InterruptedIOException: Interrupt while waiting on Operation:
> CREATE, Table Name: default:hbase_shell_tests_table, procId: 871
> 2018-10-08 02:49:09,361 INFO [Block report processor]
> blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated:
> 127.0.0.1:41338 is added to
> blk_1073742193_1369{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-ecc89143-e0a5-4a1c-b552-120be2561334:NORMAL:127.0.0.1:
>41338|RBW]]} size 58
> > TEST TIMED OUT. PRINTING THREAD DUMP. <
> {code}
> We can see that the procedure #871 wasn't stuck - the timeout cut in and
> stopped the test.
> We should separate the current test into two (or more) test files (with
> corresponding .rb) so that the execution time consistently would not exceed
> limit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[
https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652347#comment-16652347
]
Hadoop QA commented on HBASE-21073:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
57s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 8s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hbase-zookeeper in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}306m 57s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}351m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.namespace.TestNamespaceAuditor |
| | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas |
| | hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure |
| | hadoop.hbase.client.replication.TestReplicationAdmin |
| | hadoop.hbase.master.assignment.TestSplitTableRegionProcedure |
| | hadoop.hbase.replication.TestReplicationSmallTestsSync |
| | hadoop.hbase.tool.TestSecureLoadIncrementalHFiles |
| | hadoop.hbase.client.TestSnapshotDFSTemporaryDirectory |
| | hadoop.hbase.quotas.TestSpaceQuotas |
| | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.tool.TestLoadIncrementalHFiles |
| | hadoop.hbase.client.replication.TestReplicationAdminWithClusters |
| |
[jira] [Commented] (HBASE-21320) [canary] Cleanup of usage and add commentary
[ https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652321#comment-16652321 ] stack commented on HBASE-21320: --- .002 address [~psomogyi] review. Fixes tests and freshens the refguide doc. > [canary] Cleanup of usage and add commentary > > > Key: HBASE-21320 > URL: https://issues.apache.org/jira/browse/HBASE-21320 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21320.branch-2.1.001.patch, > HBASE-21320.branch-2.1.002.patch > > > I was trying to us the Canary as a diagnosis tool figuring what was online > and what was off but I couldn't make sense of how to use it. There was a bug > where we default to regionserver 'mode' always and only way to change it was > via a system property and a cryptic class name. > Changed it so we don't specify output ahead of figuring what 'mode' we are to > run in. Cleaned-up usage and added commentary. It seems useable now (Did not > change command-line args though they need changing). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21320) [canary] Cleanup of usage and add commentary
[ https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21320: -- Attachment: HBASE-21320.branch-2.1.002.patch > [canary] Cleanup of usage and add commentary > > > Key: HBASE-21320 > URL: https://issues.apache.org/jira/browse/HBASE-21320 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21320.branch-2.1.001.patch, > HBASE-21320.branch-2.1.002.patch > > > I was trying to us the Canary as a diagnosis tool figuring what was online > and what was off but I couldn't make sense of how to use it. There was a bug > where we default to regionserver 'mode' always and only way to change it was > via a system property and a cryptic class name. > Changed it so we don't specify output ahead of figuring what 'mode' we are to > run in. Cleaned-up usage and added commentary. It seems useable now (Did not > change command-line args though they need changing). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21285) Enhanced TableSnapshotInputFormat to allow a size based splitting
[
https://issues.apache.org/jira/browse/HBASE-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652319#comment-16652319
]
Ted Yu commented on HBASE-21285:
{code}
+public TableDescriptor getTableDescriptor() {
+ return delegate.getTableDescriptor();
+}
{code}
I don't see the 4 getters being used in the patch.
Do we have to expose them ?
If so, can test be added to show that ?
> Enhanced TableSnapshotInputFormat to allow a size based splitting
> -
>
> Key: HBASE-21285
> URL: https://issues.apache.org/jira/browse/HBASE-21285
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 1.4.0
>Reporter: Lavinia-Stefania Sirbu
>Assignee: Lavinia-Stefania Sirbu
>Priority: Minor
> Attachments: HBASE-21285.branch-1.4.001.patch,
> HBASE-21285.branch-1.4.002.patch, HBASE-21285.branch-1.4.003.patch,
> HBASE-21285.master.001.patch
>
>
> Currently, all the splits generated by a snapshot are having length 0. Right
> now, we have a configuration for the number of splits per region, but it's a
> general one and not very helpful when the sizes for regions are really
> different. The modification must be done in TableSnapshotInputFormatImpl
> where the length must be computed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21263) Mention compression algorithm along with other storefile details
[ https://issues.apache.org/jira/browse/HBASE-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652295#comment-16652295 ] Andrew Purtell commented on HBASE-21263: Thanks [~subrat.mishra]. I think this is fine for a pass over the code. Let me commit. Thanks for the contribution! > Mention compression algorithm along with other storefile details > > > Key: HBASE-21263 > URL: https://issues.apache.org/jira/browse/HBASE-21263 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Subrat Mishra >Priority: Minor > Labels: beginner, beginners > Attachments: HBASE-21263.master.001.patch, > HBASE-21263.master.002.patch, HBASE-21263.master.003.patch > > > Where we log storefile details we should also log the compression algorithm > used to compress blocks on disk, if any. > For example, here's a log line out of compaction: > 2018-10-02 21:59:47,594 DEBUG > [regionserver/host/1.1.1.1:8120-longCompactions-1538517461152] > compactions.Compactor: Compacting > hdfs://namenode:8020/hbase/data/default/TestTable/86037c19117a46b5b8148439ea55753b/i/3d04a7c28d6343ceb773737dbb192533, > keycount=3335873, bloomtype=ROW, size=107.5 M, encoding=ROW_INDEX_V1, > seqNum=154199, earliestPutTs=1538516084915 > Aside from bloom type, block encoding, and filename, it would be good to know > compression type in this type of DEBUG or INFO level logging. A minor > omission of information that could be helpful during debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21285) Enhanced TableSnapshotInputFormat to allow a size based splitting
[ https://issues.apache.org/jira/browse/HBASE-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lavinia-Stefania Sirbu updated HBASE-21285: --- Attachment: HBASE-21285.master.001.patch Status: Patch Available (was: Open) > Enhanced TableSnapshotInputFormat to allow a size based splitting > - > > Key: HBASE-21285 > URL: https://issues.apache.org/jira/browse/HBASE-21285 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 1.4.0 >Reporter: Lavinia-Stefania Sirbu >Assignee: Lavinia-Stefania Sirbu >Priority: Minor > Attachments: HBASE-21285.branch-1.4.001.patch, > HBASE-21285.branch-1.4.002.patch, HBASE-21285.branch-1.4.003.patch, > HBASE-21285.master.001.patch > > > Currently, all the splits generated by a snapshot are having length 0. Right > now, we have a configuration for the number of splits per region, but it's a > general one and not very helpful when the sizes for regions are really > different. The modification must be done in TableSnapshotInputFormatImpl > where the length must be computed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21285) Enhanced TableSnapshotInputFormat to allow a size based splitting
[ https://issues.apache.org/jira/browse/HBASE-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lavinia-Stefania Sirbu updated HBASE-21285: --- Status: Open (was: Patch Available) > Enhanced TableSnapshotInputFormat to allow a size based splitting > - > > Key: HBASE-21285 > URL: https://issues.apache.org/jira/browse/HBASE-21285 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 1.4.0 >Reporter: Lavinia-Stefania Sirbu >Assignee: Lavinia-Stefania Sirbu >Priority: Minor > Attachments: HBASE-21285.branch-1.4.001.patch, > HBASE-21285.branch-1.4.002.patch, HBASE-21285.branch-1.4.003.patch, > HBASE-21285.master.001.patch > > > Currently, all the splits generated by a snapshot are having length 0. Right > now, we have a configuration for the number of splits per region, but it's a > general one and not very helpful when the sizes for regions are really > different. The modification must be done in TableSnapshotInputFormatImpl > where the length must be computed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21279) Split TestAdminShell into several tests
[
https://issues.apache.org/jira/browse/HBASE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Ervits reassigned HBASE-21279:
Assignee: Artem Ervits
> Split TestAdminShell into several tests
> ---
>
> Key: HBASE-21279
> URL: https://issues.apache.org/jira/browse/HBASE-21279
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Major
> Attachments: testAdminShell-output.tar.gz
>
>
> In the flaky test board, TestAdminShell often timed out
> (https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html).
> I ran the test on Linux with SSD and reproduced the timeout (see attached
> test output).
> {code}
> 2018-10-08 02:36:09,146 DEBUG [main] hbase.HBaseTestingUtility(351): Setting
> hbase.rootdir to
> /mnt/disk2/a/2-hbase/hbase-shell/target/test-data/a103d8e4-695c-a5a9-6690-1ef2580050f9
> ...
> 2018-10-08 02:49:09,093 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=7]
> master.MasterRpcServices(1171): Checking to see if procedure is done pid=871
> Took 0.7262 seconds2018-10-08 02:49:09,324 DEBUG [PEWorker-1]
> util.FSTableDescriptors(684): Wrote into
> hdfs://localhost:43859/user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-
> d935316c9241/.tmp/data/default/hbase_shell_tests_table/.tabledesc/.tableinfo.01
> 2018-10-08 02:49:09,328 INFO
> [RegionOpenAndInitThread-hbase_shell_tests_table-1]
> regionserver.HRegion(7004): creating HRegion hbase_shell_tests_table HTD ==
> 'hbase_shell_tests_table', {NAME => 'x', VERSIONS => '5',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE =>
> 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS =>
> '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'}, {NAME => 'y', VERSIONS => '1',
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false',
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
> DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0',
> REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE =>
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false',
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE =>
> 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:43859/
> user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-d935316c9241/.tmp Table name ==
> hbase_shell_tests_table
> ^[[38;5;226mE^[[0m
> ===
> Error: ^[[48;5;16;38;5;226;1mtest_Get_simple_status(Hbase::StatusTest)^[[0m:
> Java::JavaIo::InterruptedIOException: Interrupt while waiting on Operation:
> CREATE, Table Name: default:hbase_shell_tests_table, procId: 871
> 2018-10-08 02:49:09,361 INFO [Block report processor]
> blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated:
> 127.0.0.1:41338 is added to
> blk_1073742193_1369{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-ecc89143-e0a5-4a1c-b552-120be2561334:NORMAL:127.0.0.1:
>41338|RBW]]} size 58
> > TEST TIMED OUT. PRINTING THREAD DUMP. <
> {code}
> We can see that the procedure #871 wasn't stuck - the timeout cut in and
> stopped the test.
> We should separate the current test into two (or more) test files (with
> corresponding .rb) so that the execution time consistently would not exceed
> limit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21198) Exclude dependency on net.minidev:json-smart
[
https://issues.apache.org/jira/browse/HBASE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652243#comment-16652243
]
Artem Ervits commented on HBASE-21198:
--
FWIW, [~stack], I ran individual failed tests and they all pass locally
{code:java}
mvn clean
-Dtest=org.apache.hadoop.hbase.replication.TestReplicationChangingPeerRegionservers
-Dhadoop.profile=3.0 test
mvn clean
-Dtest=org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient
-Dhadoop.profile=3.0 test
mvn clean -Dtest=org.apache.hadoop.hbase.client.TestFromClientSide#testDeletes
-Dhadoop.profile=3.0 test
mvn clean
-Dtest=org.apache.hadoop.hbase.client.TestFromClientSide#testUnmanagedHConnection
-Dhadoop.profile=3.0 test{code}
> Exclude dependency on net.minidev:json-smart
>
>
> Key: HBASE-21198
> URL: https://issues.apache.org/jira/browse/HBASE-21198
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-21198.v01.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14414/artifact/patchprocess/patch-javac-3.0.0.txt
> :
> {code}
> [ERROR] Failed to execute goal on project hbase-common: Could not resolve
> dependencies for project org.apache.hbase:hbase-common:jar:3.0.0-SNAPSHOT:
> Failed to collect dependencies at org.apache.hadoop:hadoop-common:jar:3.0.0
> -> org.apache.hadoop:hadoop-auth:jar:3.0.0 ->
> com.nimbusds:nimbus-jose-jwt:jar:4.41.1 ->
> net.minidev:json-smart:jar:2.3-SNAPSHOT: Failed to read artifact descriptor
> for net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact
> net.minidev:json-smart:pom:2.3-SNAPSHOT from/to dynamodb-local-oregon
> (https://s3-us-west-2.amazonaws.com/dynamodb-local/release): Access denied
> to:
> https://s3-us-west-2.amazonaws.com/dynamodb-local/release/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom
> , ReasonPhrase:Forbidden. -> [Help 1]
> {code}
> We should exclude dependency on net.minidev:json-smart
> hbase-common/bin/pom.xml has done so.
> The other pom.xml should do the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21310) Split TestCloneSnapshotFromClient
[
https://issues.apache.org/jira/browse/HBASE-21310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652196#comment-16652196
]
Hudson commented on HBASE-21310:
Results for branch branch-2.1
[build #474 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/474/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/474//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/474//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/474//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Split TestCloneSnapshotFromClient
> -
>
> Key: HBASE-21310
> URL: https://issues.apache.org/jira/browse/HBASE-21310
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21310-v1.patch, HBASE-21310.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21315) The getActiveMinProcId and getActiveMaxProcId of BitSetNode are incorrect if there are no active procedure
[
https://issues.apache.org/jira/browse/HBASE-21315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652197#comment-16652197
]
Hudson commented on HBASE-21315:
Results for branch branch-2.1
[build #474 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/474/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/474//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/474//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/474//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The getActiveMinProcId and getActiveMaxProcId of BitSetNode are incorrect if
> there are no active procedure
> --
>
> Key: HBASE-21315
> URL: https://issues.apache.org/jira/browse/HBASE-21315
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21315.patch
>
>
> Found by [~allan163]. If no active procedure, we will return the start as the
> min proc id, and getEnd() as the max proc id.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21320) [canary] Cleanup of usage and add commentary
[ https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652177#comment-16652177 ] stack commented on HBASE-21320: --- Thank you for the review [~psomogyi]. Fixes in next patch. > [canary] Cleanup of usage and add commentary > > > Key: HBASE-21320 > URL: https://issues.apache.org/jira/browse/HBASE-21320 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21320.branch-2.1.001.patch > > > I was trying to us the Canary as a diagnosis tool figuring what was online > and what was off but I couldn't make sense of how to use it. There was a bug > where we default to regionserver 'mode' always and only way to change it was > via a system property and a cryptic class name. > Changed it so we don't specify output ahead of figuring what 'mode' we are to > run in. Cleaned-up usage and added commentary. It seems useable now (Did not > change command-line args though they need changing). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21149) TestIncrementalBackupWithBulkLoad may fail due to file copy failure
[
https://issues.apache.org/jira/browse/HBASE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652171#comment-16652171
]
Ted Yu commented on HBASE-21149:
Added more DEBUG log:
{code}
2018-10-16 17:59:05,032 DEBUG [Time-limited test]
mapreduce.MapReduceBackupCopyJob$BackupDistCp(317):
HdfsNamedFileStatus{path=hdfs://localhost:37318/user/hbase/test-data/f61cbd60-4f59-dea6-e789-213dedb22d55/data/default/test-1539712693301/db94181f831f30089fa1ff6d0973d754/f/d0512abaaf69408b9d0edcb4c3c4cb46_SeqId_205_;
isDirectory=false; length=5100; replication=1; blocksize=134217728;
modification_time=1539712730259; access_time=1539712729843; owner=hbase;
group=supergroup; permission=rwxrwxrwx; isSymlink=false; hasAcl=false;
isEncrypted=false; isErasureCoded=false}
2018-10-16 17:59:05,033 DEBUG [Time-limited test]
mapreduce.MapReduceBackupCopyJob$BackupDistCp(317):
HdfsNamedFileStatus{path=hdfs://localhost:37318/user/hbase/test-data/f61cbd60-4f59-dea6-e789-213dedb22d55/data/default/test-1539712693301/db94181f831f30089fa1ff6d0973d754/f/fabb7b1dbe2845ca90c66d4e34987da3_SeqId_205_;
isDirectory=false; length=5142; replication=1; blocksize=134217728;
modification_time=1539712730686; access_time=1539712730274; owner=hbase;
group=supergroup; permission=rwxrwxrwx; isSymlink=false; hasAcl=false;
isEncrypted=false; isErasureCoded=false}
2018-10-16 17:59:06,222 WARN [Thread-933] mapred.LocalJobRunner$Job(590):
job_local1245512275_0004
java.io.IOException: Inconsistent sequence file: current chunk file
org.apache.hadoop.tools.CopyListingFileStatus@2bf8b9e{hdfs://localhost:37318/user/hbase/test-data/
f61cbd60-4f59-dea6-e789-213dedb22d55/data/default/test-1539712693301/db94181f831f30089fa1ff6d0973d754/f/fabb7b1dbe2845ca90c66d4e34987da3_SeqId_205_
length = 5142 aclEntries = null, xAttrs = null} doesnt match prior
entry
org.apache.hadoop.tools.CopyListingFileStatus@70f30098{hdfs://localhost:37318/user/hbase/test-data/f61cbd60-4f59-dea6-e789-213dedb22d55/data/default/test-1539712693301/db94181f831f30089fa1ff6d0973d754/f/d0512abaaf69408b9d0edcb4c3c4cb46_SeqId_205_
length = 5100 aclEntries = null, xAttrs = null}
at
org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
at
org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
{code}
We can see that the respective length of bulk loaded hfiles didn't change from
when they were recorded in the listing file to when the exception was thrown.
We should find a way to skip the concatenation performed by DictCp.
Alternatively, we can merge the bulk loaded hfiles prior to copying - but that
would require modifying the backup table which introduces more complexity in
terms of maintaining consistency for backup images.
> TestIncrementalBackupWithBulkLoad may fail due to file copy failure
> ---
>
> Key: HBASE-21149
> URL: https://issues.apache.org/jira/browse/HBASE-21149
> Project: HBase
> Issue Type: Test
> Components: backuprestore
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21149.v2.txt, HBASE-21149-v1.patch,
> testIncrementalBackupWithBulkLoad-output.txt
>
>
> From
> https://builds.apache.org/job/HBase%20Nightly/job/master/471/testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/TestIncBackupDeleteTable/
> :
> {code}
> 2018-09-03 11:54:30,526 ERROR [Time-limited test]
> impl.TableBackupClient(235): Unexpected Exception : Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> java.io.IOException: Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:351)
> at
>
[jira] [Commented] (HBASE-21320) [canary] Cleanup of usage and add commentary
[ https://issues.apache.org/jira/browse/HBASE-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652146#comment-16652146 ] Peter Somogyi commented on HBASE-21320: --- Nice cleanup [~stack]! I left some comments on RB. > [canary] Cleanup of usage and add commentary > > > Key: HBASE-21320 > URL: https://issues.apache.org/jira/browse/HBASE-21320 > Project: HBase > Issue Type: Bug > Components: canary >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21320.branch-2.1.001.patch > > > I was trying to us the Canary as a diagnosis tool figuring what was online > and what was off but I couldn't make sense of how to use it. There was a bug > where we default to regionserver 'mode' always and only way to change it was > via a system property and a cryptic class name. > Changed it so we don't specify output ahead of figuring what 'mode' we are to > run in. Cleaned-up usage and added commentary. It seems useable now (Did not > change command-line args though they need changing). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652142#comment-16652142 ] Mike Drob commented on HBASE-21073: --- v7: added another test > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch, HBASE-21073.master.003.patch, > HBASE-21073.master.004.patch, HBASE-21073.master.005.patch, > HBASE-21073.master.006.patch, HBASE-21073.master.007.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-21073: -- Attachment: HBASE-21073.master.007.patch > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch, HBASE-21073.master.003.patch, > HBASE-21073.master.004.patch, HBASE-21073.master.005.patch, > HBASE-21073.master.006.patch, HBASE-21073.master.007.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21315) The getActiveMinProcId and getActiveMaxProcId of BitSetNode are incorrect if there are no active procedure
[
https://issues.apache.org/jira/browse/HBASE-21315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652141#comment-16652141
]
Hudson commented on HBASE-21315:
Results for branch branch-2
[build #1398 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1398/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1398//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1398//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1398//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The getActiveMinProcId and getActiveMaxProcId of BitSetNode are incorrect if
> there are no active procedure
> --
>
> Key: HBASE-21315
> URL: https://issues.apache.org/jira/browse/HBASE-21315
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21315.patch
>
>
> Found by [~allan163]. If no active procedure, we will return the start as the
> min proc id, and getEnd() as the max proc id.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21310) Split TestCloneSnapshotFromClient
[
https://issues.apache.org/jira/browse/HBASE-21310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652140#comment-16652140
]
Hudson commented on HBASE-21310:
Results for branch branch-2
[build #1398 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1398/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1398//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1398//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1398//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Split TestCloneSnapshotFromClient
> -
>
> Key: HBASE-21310
> URL: https://issues.apache.org/jira/browse/HBASE-21310
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21310-v1.patch, HBASE-21310.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21315) The getActiveMinProcId and getActiveMaxProcId of BitSetNode are incorrect if there are no active procedure
[
https://issues.apache.org/jira/browse/HBASE-21315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652131#comment-16652131
]
Hudson commented on HBASE-21315:
Results for branch branch-2.0
[build #959 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/959/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/959//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/959//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/959//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> The getActiveMinProcId and getActiveMaxProcId of BitSetNode are incorrect if
> there are no active procedure
> --
>
> Key: HBASE-21315
> URL: https://issues.apache.org/jira/browse/HBASE-21315
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21315.patch
>
>
> Found by [~allan163]. If no active procedure, we will return the start as the
> min proc id, and getEnd() as the max proc id.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21310) Split TestCloneSnapshotFromClient
[
https://issues.apache.org/jira/browse/HBASE-21310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652129#comment-16652129
]
Hudson commented on HBASE-21310:
Results for branch branch-2.0
[build #959 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/959/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/959//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/959//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/959//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Split TestCloneSnapshotFromClient
> -
>
> Key: HBASE-21310
> URL: https://issues.apache.org/jira/browse/HBASE-21310
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21310-v1.patch, HBASE-21310.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21318) Make RefreshHFilesClient runnable
[
https://issues.apache.org/jira/browse/HBASE-21318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652117#comment-16652117
]
Hadoop QA commented on HBASE-21318:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
27s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
26s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 55s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
34s{color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21318 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944172/HBASE-21318.master.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux f91d08683751 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 821e4d7de2 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14716/testReport/ |
| Max. process+thread count | 2635 (vs. ulimit of 1) |
| modules | C: hbase-examples U: hbase-examples |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14716/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Make
[jira] [Updated] (HBASE-21149) TestIncrementalBackupWithBulkLoad may fail due to file copy failure
[
https://issues.apache.org/jira/browse/HBASE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21149:
---
Status: Open (was: Patch Available)
> TestIncrementalBackupWithBulkLoad may fail due to file copy failure
> ---
>
> Key: HBASE-21149
> URL: https://issues.apache.org/jira/browse/HBASE-21149
> Project: HBase
> Issue Type: Test
> Components: backuprestore
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21149.v2.txt, HBASE-21149-v1.patch,
> testIncrementalBackupWithBulkLoad-output.txt
>
>
> From
> https://builds.apache.org/job/HBase%20Nightly/job/master/471/testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/TestIncBackupDeleteTable/
> :
> {code}
> 2018-09-03 11:54:30,526 ERROR [Time-limited test]
> impl.TableBackupClient(235): Unexpected Exception : Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> java.io.IOException: Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:351)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.copyBulkLoadedFiles(IncrementalTableBackupClient.java:219)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.handleBulkLoad(IncrementalTableBackupClient.java:198)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:320)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:605)
> at
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> {code}
> However, some part of the test output was lost:
> {code}
> 2018-09-03 11:53:36,793 DEBUG [RS:0;765c9ca5ea28:36357] regions
> ...[truncated 398396 chars]...
> 8)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
