[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Adar Dembo has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12075 ) Change subject: KUDU-2636: LBM supports deleting dead containers .. KUDU-2636: LBM supports deleting dead containers this patch intends to support deleting the full containers that are dead at runtime. It can help to ease disk pressure, save tserver startup time and etc. It uses the container size and live blocks to determine whether the container is dead or live. Step1: wrapping up containers in scoped_refptr. [V] Step2: supporting dead containers removal. [V] Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Reviewed-on: http://gerrit.cloudera.org:8080/12075 Reviewed-by: Adar Dembo Tested-by: Kudu Jenkins Reviewed-by: Hao Hao --- M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h 3 files changed, 269 insertions(+), 61 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Kudu Jenkins: Verified Hao Hao: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 19 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/12075 ) Change subject: KUDU-2636: LBM supports deleting dead containers .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 18 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu Gerrit-Comment-Date: Mon, 14 Jan 2019 18:32:58 + Gerrit-HasComments: No
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12075 ) Change subject: KUDU-2636: LBM supports deleting dead containers .. Patch Set 18: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 18 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu Gerrit-Comment-Date: Sat, 12 Jan 2019 01:05:42 + Gerrit-HasComments: No
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12075 to look at the new patch set (#18). Change subject: KUDU-2636: LBM supports deleting dead containers .. KUDU-2636: LBM supports deleting dead containers this patch intends to support deleting the full containers that are dead at runtime. It can help to ease disk pressure, save tserver startup time and etc. It uses the container size and live blocks to determine whether the container is dead or live. Step1: wrapping up containers in scoped_refptr. [V] Step2: supporting dead containers removal. [V] Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 --- M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h 3 files changed, 269 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/75/12075/18 -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 18 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
helifu has posted comments on this change. (
http://gerrit.cloudera.org:8080/12075 )
Change subject: KUDU-2636: LBM supports deleting dead containers
..
Patch Set 17:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12075/17/src/kudu/fs/log_block_manager-test.cc
File src/kudu/fs/log_block_manager-test.cc:
http://gerrit.cloudera.org:8080/#/c/12075/17/src/kudu/fs/log_block_manager-test.cc@1754
PS17, Line 1754: const auto CheckMetrics = [&] (scoped_refptr&
e, const std::vector& v) {
> I think Hao's feedback about CheckMetrics vs. CheckLogMetrics from MetricsT
Done
--
To view, visit http://gerrit.cloudera.org:8080/12075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10
Gerrit-Change-Number: 12075
Gerrit-PatchSet: 17
Gerrit-Owner: helifu
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: helifu
Gerrit-Comment-Date: Sat, 12 Jan 2019 00:40:42 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Adar Dembo has posted comments on this change. (
http://gerrit.cloudera.org:8080/12075 )
Change subject: KUDU-2636: LBM supports deleting dead containers
..
Patch Set 17:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12075/17/src/kudu/fs/log_block_manager-test.cc
File src/kudu/fs/log_block_manager-test.cc:
http://gerrit.cloudera.org:8080/#/c/12075/17/src/kudu/fs/log_block_manager-test.cc@1754
PS17, Line 1754: const auto CheckMetrics = [&] (scoped_refptr&
e, const std::vector& v) {
I think Hao's feedback about CheckMetrics vs. CheckLogMetrics from MetricsTest
also applies here.
--
To view, visit http://gerrit.cloudera.org:8080/12075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10
Gerrit-Change-Number: 12075
Gerrit-PatchSet: 17
Gerrit-Owner: helifu
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: helifu
Gerrit-Comment-Date: Fri, 11 Jan 2019 23:56:07 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12075 to look at the new patch set (#17). Change subject: KUDU-2636: LBM supports deleting dead containers .. KUDU-2636: LBM supports deleting dead containers this patch intends to support deleting the full containers that are dead at runtime. It can help to ease disk pressure, save tserver startup time and etc. It uses the container size and live blocks to determine whether the container is dead or live. Step1: wrapping up containers in scoped_refptr. [V] Step2: supporting dead containers removal. [V] Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 --- M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h 3 files changed, 238 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/75/12075/17 -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 17 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/12075 ) Change subject: KUDU-2636: LBM supports deleting dead containers .. Patch Set 16: (2 comments) http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager-test.cc File src/kudu/fs/log_block_manager-test.cc: http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager-test.cc@311 PS12, Line 311: 0, 0, 1, 0, 0, 0, 0 > Yeah, in that case, could you please rollback the change, otherwise it is r Done http://gerrit.cloudera.org:8080/#/c/12075/16/src/kudu/fs/log_block_manager-test.cc File src/kudu/fs/log_block_manager-test.cc: http://gerrit.cloudera.org:8080/#/c/12075/16/src/kudu/fs/log_block_manager-test.cc@1725 PS16, Line 1725: i+1 > nit: 'i + 1' Done -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 16 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 11 Jan 2019 23:23:17 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12075 ) Change subject: KUDU-2636: LBM supports deleting dead containers .. Patch Set 16: > All the tests passed locally except ASAN. Failing on an unrelated test case > is weird. Let me retrigger the build. Yeah that's an unrelated failure that Andrew Wong has been looking into. Don't worry about it. -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 16 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 11 Jan 2019 20:03:45 + Gerrit-HasComments: No
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/12075 ) Change subject: KUDU-2636: LBM supports deleting dead containers .. Patch Set 16: Code-Review+1 (2 comments) Just some minor nits. http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager-test.cc File src/kudu/fs/log_block_manager-test.cc: http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager-test.cc@311 PS12, Line 311: 0, 0, 1, 0, 0, 0, 0 > Sorry, that's not a good solution i think. My motivation for the changes he Yeah, in that case, could you please rollback the change, otherwise it is really easy to lost tracking of what does the number means. http://gerrit.cloudera.org:8080/#/c/12075/16/src/kudu/fs/log_block_manager-test.cc File src/kudu/fs/log_block_manager-test.cc: http://gerrit.cloudera.org:8080/#/c/12075/16/src/kudu/fs/log_block_manager-test.cc@1725 PS16, Line 1725: i+1 nit: 'i + 1' -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 16 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 11 Jan 2019 18:10:03 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/12075 ) Change subject: KUDU-2636: LBM supports deleting dead containers .. Patch Set 15: All the tests passed locally except ASAN. Failing on an unrelated test case is weird. Let me retrigger the build. -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 15 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 11 Jan 2019 12:20:16 + Gerrit-HasComments: No
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12075 to look at the new patch set (#16). Change subject: KUDU-2636: LBM supports deleting dead containers .. KUDU-2636: LBM supports deleting dead containers this patch intends to support deleting the full containers that are dead at runtime. It can help to ease disk pressure, save tserver startup time and etc. It uses the container size and live blocks to determine whether the container is dead or live. Step1: wrapping up containers in scoped_refptr. [V] Step2: supporting dead containers removal. [V] Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 --- M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h 3 files changed, 241 insertions(+), 121 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/75/12075/16 -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 16 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/12075 ) Change subject: KUDU-2636: LBM supports deleting dead containers .. Patch Set 15: I'm sorry to have omitted this case! :( -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 15 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 11 Jan 2019 11:03:29 + Gerrit-HasComments: No
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12075 ) Change subject: KUDU-2636: LBM supports deleting dead containers .. Patch Set 15: Code-Review+2 Leaving unmerged for Hao to take another look. -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 15 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu Gerrit-Comment-Date: Fri, 11 Jan 2019 04:00:45 + Gerrit-HasComments: No
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
helifu has posted comments on this change. (
http://gerrit.cloudera.org:8080/12075 )
Change subject: KUDU-2636: LBM supports deleting dead containers
..
Patch Set 14:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager.cc
File src/kudu/fs/log_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager.cc@760
PS12, Line 760: {
: uint64_t metadata_size = 0
> Oh, I think I get you now. It took a second reading of the code for me to s
I defer to your opinion.
http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager.cc@2366
PS12, Line 2366: } else if (static_cast(container->live_blocks()) /
: container->total_blocks() <=
FLAGS_log_container_live_metadata_before_compact_ratio) {
: // Metadata files of containers with very few live
blocks will be compacte
> I actually think we should keep this TODO as-is. Yes, we support container
Done
--
To view, visit http://gerrit.cloudera.org:8080/12075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10
Gerrit-Change-Number: 12075
Gerrit-PatchSet: 14
Gerrit-Owner: helifu
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: helifu
Gerrit-Comment-Date: Fri, 11 Jan 2019 01:37:02 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12075 to look at the new patch set (#15). Change subject: KUDU-2636: LBM supports deleting dead containers .. KUDU-2636: LBM supports deleting dead containers this patch intends to support deleting the full containers that are dead at runtime. It can help to ease disk pressure, save tserver startup time and etc. It uses the container size and live blocks to determine whether the container is dead or live. Step1: wrapping up containers in scoped_refptr. [V] Step2: supporting dead containers removal. [V] Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 --- M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h 3 files changed, 236 insertions(+), 121 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/75/12075/15 -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 15 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Adar Dembo has posted comments on this change. (
http://gerrit.cloudera.org:8080/12075 )
Change subject: KUDU-2636: LBM supports deleting dead containers
..
Patch Set 14:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager.cc
File src/kudu/fs/log_block_manager.cc:
http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager.cc@760
PS12, Line 760: {
: uint64_t metadata_size = 0
> The motivation here is to support deleting incomplete files directly, other
Oh, I think I get you now. It took a second reading of the code for me to spot
the subtle difference: if one container file is missing, we'll delete the other
file, even if it has a non-zero file size. Previously we would only delete the
other file if it was empty (data file) or under the minimum length (metadata
file).
Right, it's possible for an ill-timed host crash to leave behind a non-empty
data file (containing for example one block) and either an empty or
non-existent metadata file. The concern though is, if some other event caused
one of the data or metadata file to disappear, by deleting the other file we'd
be making manual recovery impossible. Meaning, Kudu's well-intentioned action
(to automate recovery by deleting the other container file) could in fact be
leading to real data loss.
I'm open to being convinced otherwise, but that's the position I currently have
on this issue.
http://gerrit.cloudera.org:8080/#/c/12075/12/src/kudu/fs/log_block_manager.cc@2366
PS12, Line 2366: } else if (static_cast(container->live_blocks()) /
: container->total_blocks() <=
FLAGS_log_container_live_metadata_before_compact_ratio) {
: // Metadata files of containers with very few live
blocks will be compacte
> Done
I actually think we should keep this TODO as-is. Yes, we support container
deletion at realtime now, but this change isn't actually addressing the TODO,
so the request in it is still valid and worth preserving.
--
To view, visit http://gerrit.cloudera.org:8080/12075
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10
Gerrit-Change-Number: 12075
Gerrit-PatchSet: 14
Gerrit-Owner: helifu
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: helifu
Gerrit-Comment-Date: Thu, 10 Jan 2019 21:51:54 +
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2636: LBM supports deleting dead containers
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Hao Hao, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12075 to look at the new patch set (#14). Change subject: KUDU-2636: LBM supports deleting dead containers .. KUDU-2636: LBM supports deleting dead containers this patch intends to support deleting the full containers that are dead at runtime. It can help to ease disk pressure, save tserver startup time and etc. It uses the container size and live blocks to determine whether the container is dead or live. Step1: wrapping up containers in scoped_refptr. [V] Step2: supporting dead containers removal. [V] Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 --- M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h 3 files changed, 236 insertions(+), 125 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/75/12075/14 -- To view, visit http://gerrit.cloudera.org:8080/12075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10 Gerrit-Change-Number: 12075 Gerrit-PatchSet: 14 Gerrit-Owner: helifu Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: helifu
