[jira] [Comment Edited] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[ https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577029#comment-16577029 ] Tak Lon (Stephen) Wu edited comment on HBASE-21011 at 8/11/18 5:08 AM: --- Thanks [~reidchan] for your suggestion about using {{hdfs -rm}} but I don't think that would be a good suggestion to operator due to any error-prone entering (pointing to directory other than oldWALs or HFiles archive directory) while using {{-rm}}. I hope not but it could lead to data loss disaster (that's why I don't suggest operator to use hdfs command as workaround if possible). But I do agree with you about the point of `endless requirement`, once I get back from our use cases, I will close this item. Also, I think part of the HBase on Cloud should be related HBASE-20952 about WAL interface decouple from HDFS (although I didn't put any comment), but IMHO backward compatibility should not only related this {{run_cleaner_chore}} admin CLI but all related components that handles WAL. Anyway, I will keep this in mind for future improvements/changes related to WAL. was (Author: taklwu): Thanks [~reidchan] for your suggestion about using {{hdfs -rm}} but I don't think that would be a good suggestion to operator due to any error-prone entering while using {{-rm}} could lead to data loss disaster. But I do agree with you about the point of `endless requirement`, once I get back from our use cases, I will close this item. Also, I think part of the HBase on Cloud should be related HBASE-20952 about WAL interface decouple from HDFS (although I didn't put any comment), but IMHO backward compatibility should not only related this {{run_cleaner_chore}} admin CLI but all related components that handles WAL. Anyway, I will keep this in mind for future improvements/changes related to WAL. > Provide CLI option to run oldwals and hfiles cleaner separately when cleaner > chore is disabled > -- > > Key: HBASE-21011 > URL: https://issues.apache.org/jira/browse/HBASE-21011 > Project: HBase > Issue Type: Improvement > Components: Admin, Client >Affects Versions: 3.0.0, 1.4.6, 2.1.1 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Attachments: HBASE-21011.master.001.patch, > HBASE-21011.master.002.patch, HBASE-21011.master.003.patch, > HBASE-21011.master.004.patch > > > There is a corner case when cleaner chore for HFiles and oldwals is disabled, > admin/user needs to manually execute admin command {{cleaner_chore_run}} to > clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is > to [firstly trigger the HFiles cleaner and then oldwals > cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420], > and only return succeed if both completes. > but when running this {{cleaner_chore_run}} command, there is a potential use > case that admin would like trigger the cleaner for only oldwals or hfiles but > still keep the automatic cleaner chore disabled. So, this change aims to > provide support for this corner case, and provide flexibility for those user > with cleaner chore disabled by default to execute admin CLI to run oldwals > and HFiles cleaning procedure individually. > NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch > added options 'hfiles' and 'oldwals' to it. Also fix default behavior of > {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled, > e.g. the proposed admin CLI options are > {noformat} > hbase> cleaner_chore_run # this was introduced in HBASE-17280, > but changed the behavior to only ran when cleaner chore is set to disabled > hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set > to disabled > hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set > to disabled > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[ https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577042#comment-16577042 ] Reid Chan edited comment on HBASE-21011 at 8/11/18 4:02 AM: I stand my point, and vote won't change. Please ensure it has at least two +1s from other committers before it goes in any branch, since there's a -1. was (Author: reidchan): I stand my point, and vote won't change. Please ensure it has at least two {{+1}}s from other committers before it goes in any branch, since there's a {{-1}}. :) > Provide CLI option to run oldwals and hfiles cleaner separately when cleaner > chore is disabled > -- > > Key: HBASE-21011 > URL: https://issues.apache.org/jira/browse/HBASE-21011 > Project: HBase > Issue Type: Improvement > Components: Admin, Client >Affects Versions: 3.0.0, 1.4.6, 2.1.1 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Attachments: HBASE-21011.master.001.patch, > HBASE-21011.master.002.patch, HBASE-21011.master.003.patch, > HBASE-21011.master.004.patch > > > There is a corner case when cleaner chore for HFiles and oldwals is disabled, > admin/user needs to manually execute admin command {{cleaner_chore_run}} to > clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is > to [firstly trigger the HFiles cleaner and then oldwals > cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420], > and only return succeed if both completes. > but when running this {{cleaner_chore_run}} command, there is a potential use > case that admin would like trigger the cleaner for only oldwals or hfiles but > still keep the automatic cleaner chore disabled. So, this change aims to > provide support for this corner case, and provide flexibility for those user > with cleaner chore disabled by default to execute admin CLI to run oldwals > and HFiles cleaning procedure individually. > NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch > added options 'hfiles' and 'oldwals' to it. Also fix default behavior of > {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled, > e.g. the proposed admin CLI options are > {noformat} > hbase> cleaner_chore_run # this was introduced in HBASE-17280, > but changed the behavior to only ran when cleaner chore is set to disabled > hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set > to disabled > hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set > to disabled > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[ https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570992#comment-16570992 ] Andrew Purtell edited comment on HBASE-21011 at 8/7/18 1:05 AM: Because this changes on the wire messages we need to consider wire compatibility. I don't think this will work: {code:java} message RunCleanerChoreRequest { + required CleanerType type = 1; } {code} Instead, CleanerType should be optional with a default value of DEFAULT. Otherwise newer masters will choke on requests from older admin clients that will not provide 'type' was (Author: apurtell): Because this changes on the wire messages we need to consider wire compatibility. I don't think this will work: {code:java} message RunCleanerChoreRequest { + required CleanerType type = 1; } {code} Instead, CleanerType should be optional with a default value of DEFAULT. Otherwise newer masters will choke on requests from older admin clients that will not provide 'type' Definitely needed if considering this for commit to branch-1. > Provide CLI option to run oldwals and hfiles cleaner separately when cleaner > chore is disabled > -- > > Key: HBASE-21011 > URL: https://issues.apache.org/jira/browse/HBASE-21011 > Project: HBase > Issue Type: Improvement > Components: Admin, Client >Affects Versions: 3.0.0, 1.4.6, 2.1.1 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Attachments: HBASE-21011.master.001.patch, > HBASE-21011.master.002.patch > > > There is a corner case when cleaner chore for HFiles and oldwals is disabled, > admin/user needs to manually execute admin command {{cleaner_chore_run}} to > clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is > to [firstly trigger the HFiles cleaner and then oldwals > cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420], > and only return succeed if both completes. > but when running this {{cleaner_chore_run}} command, there is a potential use > case that admin would like trigger the cleaner for only oldwals or hfiles but > still keep the automatic cleaner chore disabled. So, this change aims to > provide support for this corner case, and provide flexibility for those user > with cleaner chore disabled by default to execute admin CLI to run oldwals > and HFiles cleaning procedure individually. > NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch > added options 'hfiles' and 'oldwals' to it. Also fix default behavior of > {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled, > e.g. the proposed admin CLI options are > {noformat} > hbase> cleaner_chore_run # this was introduced in HBASE-17280, > but changed the behavior to only ran when cleaner chore is set to disabled > hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set > to disabled > hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set > to disabled > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[ https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569360#comment-16569360 ] Tak Lon (Stephen) Wu edited comment on HBASE-21011 at 8/5/18 4:33 AM: -- [~reidchan] thanks for the vote, if we see another -1 vote on this issue, I will discard the patch :P. In addition, I rethought a bit on this issue with your suggestion, I will probably update the PR/patch for {{cleaner_chore_run}}/{{Admin#runCleanerChore}} (or in a different JIRA) should only be executed when {{CleanerChore}} for {{LogCleaner}} and {{HFileCleaner}} are disabled. That should make senses to the operator and CLI. was (Author: taklwu): [~reidchan] thanks for the vote, if we see another -1 vote on this issue, I will discard the patch :P > Provide CLI option to run oldwals and hfiles cleaner separately when cleaner > chore is disabled > -- > > Key: HBASE-21011 > URL: https://issues.apache.org/jira/browse/HBASE-21011 > Project: HBase > Issue Type: Improvement > Components: Admin, Client >Affects Versions: 3.0.0, 1.4.6, 2.1.1 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Fix For: 3.0.0 > > > There is a corner case when cleaner chore for HFiles and oldwals is disabled, > admin/user needs to manually execute admin command {{cleaner_chore_run}} to > clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is > to [firstly trigger the HFiles cleaner and then oldwals > cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420], > and only return succeed if both completes. > but when running this {{cleaner_chore_run}} command, there is a potential use > case that admin would like trigger the cleaner for only oldwals or hfiles but > still keep the automatic cleaner chore disabled. So, this change aims to > provide support for this corner case, and provide flexibility for those user > with cleaner chore disabled by default to execute admin CLI to run oldwals > and HFiles cleaning procedure individually. > NOTE that we keep the default as running both of them for backward > compatibility, e.g. the proposed admin CLI options are > {noformat} > hbase> cleaner_chore_run > hbase> cleaner_chore_run 'hfiles' > hbase> cleaner_chore_run 'oldwals' > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)