[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12070: -- Fix Version/s: (was: 1.0.1) 1.0.0 > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.0.1, 1.1.0, 0.98.11 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 1.1.0, 0.98.11 > > Attachments: HBASE-12070.v1-0.98.patch, > HBASE-12070.v1-branch-1.patch, HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Fix Version/s: 0.98.11 > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.0.1, 1.1.0, 0.98.11 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.0.1, 1.1.0, 0.98.11 > > Attachments: HBASE-12070.v1-0.98.patch, > HBASE-12070.v1-branch-1.patch, HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Affects Version/s: 0.98.11 > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.0.1, 1.1.0, 0.98.11 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.0.1, 1.1.0, 0.98.11 > > Attachments: HBASE-12070.v1-0.98.patch, > HBASE-12070.v1-branch-1.patch, HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Affects Version/s: 1.0.1 > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.0.1, 1.1.0, 0.98.11 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.0.1, 1.1.0, 0.98.11 > > Attachments: HBASE-12070.v1-0.98.patch, > HBASE-12070.v1-branch-1.patch, HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Attachment: HBASE-12070.v1-0.98.patch 0.98 patch is available. > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.0.1, 1.1.0 > > Attachments: HBASE-12070.v1-0.98.patch, > HBASE-12070.v1-branch-1.patch, HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12070: -- Resolution: Fixed Fix Version/s: 1.0.1 Release Note: Adds a new option to HBCK tool -fixOrphanedTableZnodes, which fixes orphaned table entries in zookeeper which does not have corresponding meta entries. This state can be from a failed create table attempt. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to branch-1 and branch-1.0. We don't need it in master since zk table state is gone in master, and it moves to meta. Further ProcV2 will remove the need. Thanks Stephen for the patch. [~apurtell] do you want this in 0.98? > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.0.1, 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch, > HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Status: In Progress (was: Patch Available) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch, > HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Status: Patch Available (was: In Progress) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch, > HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Attachment: HBASE-12070.v2-branch-1.patch > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch, > HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Status: In Progress (was: Patch Available) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch, > HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Status: Patch Available (was: In Progress) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch, > HBASE-12070.v2-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Status: Patch Available (was: In Progress) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Attachment: HBASE-12070.v1-branch-1.patch > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Attachment: (was: HBASE-12070.v1-branch-1.patch) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Status: In Progress (was: Patch Available) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12070: --- Status: Patch Available (was: Open) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Attachment: HBASE-12070.v1-branch-1.patch > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Attachment: (was: HBASE-12070.v1-branch-1.patch) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Attachment: HBASE-12070.v1-branch-1.patch > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Attachment: (was: HBASE-12070.v1-branch-1.patch) > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Affects Version/s: 1.1.0 > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Attachment: HBASE-12070.v1-branch-1.patch The branch-1 patch is attached. Due to major refactoring in master branch (and major improvement planned in this area), this patch is NOT applicable to master branch > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Fix Version/s: 1.1.0 > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 1.1.0 > > Attachments: HBASE-12070.v1-branch-1.patch > > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Component/s: hbck > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug > Components: hbck >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12070) Add an option to hbck to fix ZK inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12070: --- Assignee: Stephen Yuan Jiang > Add an option to hbck to fix ZK inconsistencies > --- > > Key: HBASE-12070 > URL: https://issues.apache.org/jira/browse/HBASE-12070 > Project: HBase > Issue Type: Bug >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > > If the HMaster bounces in the middle of table creation, we could be left in a > state where a znode exists for the table, but that hasn't percolated into > META or to HDFS. We've run into this a couple times on our clusters. Once the > table is in this state, the only fix is to rm the znode using the > zookeeper-client. Doing this manually looks a bit error prone. Could an > option be added to hbck to catch and fix such inconsistencies? > A more general issue I'd like comment on is whether it makes sense for > HMaster to be maintaining its own write-ahead log? The idea would be that on > a bounce, the master would discover it was in the middle of creating a table > and either rollback or complete that operation? An issue that we observed > recently was that a table that was in DISABLING state before a bounce was not > in that state after. A write-ahead log to persist table state changes seems > useful. Now, all of this state could be in ZK instead of the WAL - it doesn't > matter where it gets persisted as long as it does. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
