[kudu-CR] [Tool] Find the container by block id
Wang Xixu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20594 Change subject: [Tool] Find the container by block id .. [Tool] Find the container by block id Sometimes, a block is damaged because of the checksum is incorrect or the disk is damaged. The log only prints the block id, but no container id. We want to known which container the block locates in. If we known the container, we can reads the related metadata file to get the status of the block: dead or alive. We can also truncate the record from the meadata file to repair the container, although it will cause some data loss. This patch provides the tool to find the container by a block id. Change-Id: Ifaafd2bb634ed7fab923d9cf9eef40437442d187 --- M src/kudu/fs/block_manager.h M src/kudu/fs/file_block_manager.cc M src/kudu/fs/file_block_manager.h M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_local_replica.cc 7 files changed, 119 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/20594/1 -- To view, visit http://gerrit.cloudera.org:8080/20594 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifaafd2bb634ed7fab923d9cf9eef40437442d187 Gerrit-Change-Number: 20594 Gerrit-PatchSet: 1 Gerrit-Owner: Wang Xixu <1450306...@qq.com>
[kudu-CR] [Tool] Find the container by block id
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20594 to look at the new patch set (#2). Change subject: [Tool] Find the container by block id .. [Tool] Find the container by block id Sometimes, a block is damaged because of the checksum is incorrect or the disk is damaged. The log only prints the block id, but no container id. We want to known which container the block locates in. If we known the container, we can reads the related metadata file to get the status of the block: dead or alive. We can also truncate the record from the meadata file to repair the container, although it will cause some data loss. This patch provides the tool to find the container by a block id. Change-Id: Ifaafd2bb634ed7fab923d9cf9eef40437442d187 --- M src/kudu/fs/block_manager.h M src/kudu/fs/file_block_manager.cc M src/kudu/fs/file_block_manager.h M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_local_replica.cc 7 files changed, 119 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/20594/2 -- To view, visit http://gerrit.cloudera.org:8080/20594 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifaafd2bb634ed7fab923d9cf9eef40437442d187 Gerrit-Change-Number: 20594 Gerrit-PatchSet: 2 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [compaction] Skip memory allocation for ancient undo deltas
Ashwani Raina has posted comments on this change. ( http://gerrit.cloudera.org:8080/20546 ) Change subject: [compaction] Skip memory allocation for ancient undo deltas .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/20546/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20546/1//COMMIT_MSG@7 PS1, Line 7: [compaction] Skip memory allocation for ancient undo deltas > I think it's necessary to add a test at least to verify that the data is st Thanks for the review, Alexey. Apologies for being late on this. There is one test (TestNoGenerateUndoOnMajorDeltaCompaction) that does cover this code path already. It updates rows couple of times and then flushes the DMS thereby creating REDO deltas for those updates. Later, it moves the clock forwards to make those deltas ancient. Subsequent reads on the rows are verified for both the updates. Lastly, major delta compaction is run that results in this execution of exactly this piece of code where REDO mutations are processed and converted to UNDO deltas. However, in this specific case, we end with AHM case because by now all these deltas are ancient. So, what we have eventually is this: Base row data filled with latest update and no REDO mutations and just one UNDO delta for DELETE for first insert op. The only difference here is that REDO was applied from major delta compaction instead of rowset compaction. Since ApplyMutationsAndGenerateUndos is applicable to both Major and RowSet compaction, it doesn't make much difference except that major compaction won't touch the already generated UNDO deltas as it only works on currently existing REDO deltas and applies those to base row. However, rowset compaction does take into consideration existing UNDO deltas which in this specific test case would be only 1 in number (for the first insert). Either way, we are only working on the existing REDO deltas here and make sure that we don't allocate memory for their corresponding in-memory UNDO delta object if the REDO mutation is ancient. So, in a nutshell, the existing test already verifies that there is no UNDO delta created for ancient REDO mutations and also make sure that those REDO updates are applied to the base row thereby verifying that correct data is applied on the row. I hope my explanation above didn't end up creating more confusion :-) If you want, I can add one similar test that does the similar steps and verify at the end the following: 1. There is no ancient UNDO delta as well as REDO delta present. 2. The row has latest value applied to it. If you have something different and extra in mind with regard to fuzz tests, I am open to discussion. -- To view, visit http://gerrit.cloudera.org:8080/20546 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb41636a3ac063478fe181560d4ec85dadeb0ef3 Gerrit-Change-Number: 20546 Gerrit-PatchSet: 1 Gerrit-Owner: Ashwani Raina Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Thu, 19 Oct 2023 11:51:05 + Gerrit-HasComments: Yes
[kudu-CR] Small refactor to use DirManager::WaitOnClosures()
Yingchun Lai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20596 Change subject: Small refactor to use DirManager::WaitOnClosures() .. Small refactor to use DirManager::WaitOnClosures() Change-Id: Ib2f2ca771f444ac149c7f27ef364c07c8efbb8c7 --- M src/kudu/fs/log_block_manager-test.cc 1 file changed, 3 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/96/20596/1 -- To view, visit http://gerrit.cloudera.org:8080/20596 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib2f2ca771f444ac149c7f27ef364c07c8efbb8c7 Gerrit-Change-Number: 20596 Gerrit-PatchSet: 1 Gerrit-Owner: Yingchun Lai
[kudu-CR] [compaction] Skip memory allocation for ancient undo deltas
Hello Tidy Bot, Alexey Serbin, Zoltan Martonka, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20546 to look at the new patch set (#2). Change subject: [compaction] Skip memory allocation for ancient undo deltas .. [compaction] Skip memory allocation for ancient undo deltas Currently, while applying REDO mutations to base row and create corresponding UNDO deltas for each REDO mutation, we don't check whether any mutation is ancient that is anyway going to be ignored and removed from list of UNDO deltas in later stage of processing. This change checks each REDO mutation beforehand and doesn't allocate any memory if found ancient. This avoids unnecessary memory usage. Change-Id: Ibb41636a3ac063478fe181560d4ec85dadeb0ef3 --- M src/kudu/tablet/compaction.cc M src/kudu/tablet/compaction.h M src/kudu/tablet/delta_compaction.cc M src/kudu/tablet/tablet-test-base.h M src/kudu/tablet/tablet_history_gc-test.cc 5 files changed, 121 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/20546/2 -- To view, visit http://gerrit.cloudera.org:8080/20546 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibb41636a3ac063478fe181560d4ec85dadeb0ef3 Gerrit-Change-Number: 20546 Gerrit-PatchSet: 2 Gerrit-Owner: Ashwani Raina Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Zoltan Martonka
[kudu-CR] Small refactor to use DirManager::WaitOnClosures()
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20596 ) Change subject: Small refactor to use DirManager::WaitOnClosures() .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20596 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2f2ca771f444ac149c7f27ef364c07c8efbb8c7 Gerrit-Change-Number: 20596 Gerrit-PatchSet: 1 Gerrit-Owner: Yingchun Lai Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai Gerrit-Reviewer: Yuqi Du Gerrit-Comment-Date: Thu, 19 Oct 2023 16:35:26 + Gerrit-HasComments: No
[kudu-CR] Small refactor to use DirManager::WaitOnClosures()
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20596 ) Change subject: Small refactor to use DirManager::WaitOnClosures() .. Small refactor to use DirManager::WaitOnClosures() Change-Id: Ib2f2ca771f444ac149c7f27ef364c07c8efbb8c7 Reviewed-on: http://gerrit.cloudera.org:8080/20596 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin --- M src/kudu/fs/log_block_manager-test.cc 1 file changed, 3 insertions(+), 9 deletions(-) Approvals: Kudu Jenkins: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20596 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib2f2ca771f444ac149c7f27ef364c07c8efbb8c7 Gerrit-Change-Number: 20596 Gerrit-PatchSet: 2 Gerrit-Owner: Yingchun Lai Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai Gerrit-Reviewer: Yuqi Du
[kudu-CR] WIP Make --encrypt data at rest configurable for kudu-tool-test
Yingchun Lai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20597 Change subject: WIP Make --encrypt_data_at_rest configurable for kudu-tool-test .. WIP Make --encrypt_data_at_rest configurable for kudu-tool-test Change-Id: I5cba40850d2d2ef884f45e252b8117ae12b8f9d9 --- M src/kudu/tools/kudu-tool-test.cc 1 file changed, 185 insertions(+), 209 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/97/20597/1 -- To view, visit http://gerrit.cloudera.org:8080/20597 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5cba40850d2d2ef884f45e252b8117ae12b8f9d9 Gerrit-Change-Number: 20597 Gerrit-PatchSet: 1 Gerrit-Owner: Yingchun Lai
[kudu-CR](gh-pages) [site] Update slack invite link - take 2
Abhishek Chennaka has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20598 Change subject: [site] Update slack invite link - take 2 .. [site] Update slack invite link - take 2 This commit updates the broken slack invite link in all the locations where it is being used. This link works for 400 users after which it needs to be regenerated and updated. Change-Id: Id3090bcb452c72873fedd32b2b3074ebd99b3f87 --- M _includes/top_common.html M _posts/2018-08-06-getting-started-with-kudu-an-oreilly-title.md M faq.md 3 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/20598/1 -- To view, visit http://gerrit.cloudera.org:8080/20598 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: newchange Gerrit-Change-Id: Id3090bcb452c72873fedd32b2b3074ebd99b3f87 Gerrit-Change-Number: 20598 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Chennaka
[kudu-CR](gh-pages) [site] Update slack invite link - take 2
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20598 ) Change subject: [site] Update slack invite link - take 2 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20598 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: Id3090bcb452c72873fedd32b2b3074ebd99b3f87 Gerrit-Change-Number: 20598 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Comment-Date: Thu, 19 Oct 2023 16:56:47 + Gerrit-HasComments: No
[kudu-CR] refactor: move SetEncryptionFlags to constructors
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20592 ) Change subject: refactor: move SetEncryptionFlags to constructors .. Patch Set 1: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/20592/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20592/1//COMMIT_MSG@9 PS1, Line 9: util nit: utility http://gerrit.cloudera.org:8080/#/c/20592/1//COMMIT_MSG@11 PS1, Line 11: parameterize nit: parameterized -- To view, visit http://gerrit.cloudera.org:8080/20592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67dca5a45f78de87309d0434c3b003a83f28e1a6 Gerrit-Change-Number: 20592 Gerrit-PatchSet: 1 Gerrit-Owner: Yingchun Lai Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Thu, 19 Oct 2023 17:48:00 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-3407 Avoid unchecked scheduling of flush operations.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20166 ) Change subject: KUDU-3407 Avoid unchecked scheduling of flush operations. .. Patch Set 31: > Build Failed > > http://jenkins.kudu.apache.org/job/kudu-gerrit/28625/ : FAILURE LINT isn't yet happy, but the rest seem to be unrelated test failures (at least a first glance). As for the LINT, please address the following: 04:56:46 /home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/maintenance_manager-test.cc:354: Lines should very rarely be longer than 100 characters [whitespace/line_length] [4] 04:56:46 Total errors found: 1 -- To view, visit http://gerrit.cloudera.org:8080/20166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc2fd3a850cf99d54ef2980211b712468440ed80 Gerrit-Change-Number: 20166 Gerrit-PatchSet: 31 Gerrit-Owner: Song Jiacheng Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Song Jiacheng Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Comment-Date: Thu, 19 Oct 2023 17:50:04 + Gerrit-HasComments: No
[kudu-CR] [test] fix negotiation-test for krb5 version 1.19.2
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20599 Change subject: [test] fix negotiation-test for krb5 version 1.19.2 .. [test] fix negotiation-test for krb5 version 1.19.2 TestNegotiation.TestPreflight was failing on Ubuntu 22.04.3 LTS because of the error message mismatch: src/kudu/rpc/negotiation-test.cc:1533: Failure Value of: s.ToString() Expected: contains regular expression "error accessing keytab: Permission denied " Actual: "Runtime error: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Permission denied)" This patch addresses the issue, making the reference error message pattern more generic. Change-Id: If4a3772fa50fc799756da121644c9dbb4d9bc977 --- M src/kudu/rpc/negotiation-test.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/99/20599/1 -- To view, visit http://gerrit.cloudera.org:8080/20599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If4a3772fa50fc799756da121644c9dbb4d9bc977 Gerrit-Change-Number: 20599 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin
[kudu-CR] [test] fix negotiation-test for krb5 version 1.19.2
Alexey Serbin has removed a vote on this change. Change subject: [test] fix negotiation-test for krb5 version 1.19.2 .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/20599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: If4a3772fa50fc799756da121644c9dbb4d9bc977 Gerrit-Change-Number: 20599 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai
[kudu-CR] [test] fix negotiation-test for krb5 version 1.19.2
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/20599 ) Change subject: [test] fix negotiation-test for krb5 version 1.19.2 .. Patch Set 1: Verified+1 unrelated test failure in TlsSocketTest.TestRecvFailure (DEBUG) -- To view, visit http://gerrit.cloudera.org:8080/20599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4a3772fa50fc799756da121644c9dbb4d9bc977 Gerrit-Change-Number: 20599 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Fri, 20 Oct 2023 00:03:02 + Gerrit-HasComments: No
[kudu-CR] [test] fix negotiation-test for krb5 version 1.19.2
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/20599 ) Change subject: [test] fix negotiation-test for krb5 version 1.19.2 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4a3772fa50fc799756da121644c9dbb4d9bc977 Gerrit-Change-Number: 20599 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Fri, 20 Oct 2023 00:04:22 + Gerrit-HasComments: No
[kudu-CR] [test] fix negotiation-test for krb5 version 1.19.2
Abhishek Chennaka has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20599 ) Change subject: [test] fix negotiation-test for krb5 version 1.19.2 .. [test] fix negotiation-test for krb5 version 1.19.2 TestNegotiation.TestPreflight was failing on Ubuntu 22.04.3 LTS because of the error message mismatch: src/kudu/rpc/negotiation-test.cc:1533: Failure Value of: s.ToString() Expected: contains regular expression "error accessing keytab: Permission denied " Actual: "Runtime error: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Permission denied)" This patch addresses the issue, making the reference error message pattern more generic. Change-Id: If4a3772fa50fc799756da121644c9dbb4d9bc977 Reviewed-on: http://gerrit.cloudera.org:8080/20599 Tested-by: Alexey Serbin Reviewed-by: Abhishek Chennaka --- M src/kudu/rpc/negotiation-test.cc 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Alexey Serbin: Verified Abhishek Chennaka: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/20599 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If4a3772fa50fc799756da121644c9dbb4d9bc977 Gerrit-Change-Number: 20599 Gerrit-PatchSet: 2 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai
[kudu-CR](gh-pages) [site] Update slack invite link - take 2
Abhishek Chennaka has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20598 ) Change subject: [site] Update slack invite link - take 2 .. [site] Update slack invite link - take 2 This commit updates the broken slack invite link in all the locations where it is being used. This link works for 400 users after which it needs to be regenerated and updated. Change-Id: Id3090bcb452c72873fedd32b2b3074ebd99b3f87 Reviewed-on: http://gerrit.cloudera.org:8080/20598 Reviewed-by: Alexey Serbin Tested-by: Abhishek Chennaka --- M _includes/top_common.html M _posts/2018-08-06-getting-started-with-kudu-an-oreilly-title.md M faq.md 3 files changed, 3 insertions(+), 3 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved Abhishek Chennaka: Verified -- To view, visit http://gerrit.cloudera.org:8080/20598 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: merged Gerrit-Change-Id: Id3090bcb452c72873fedd32b2b3074ebd99b3f87 Gerrit-Change-Number: 20598 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Chennaka Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin
[kudu-CR](gh-pages) [site] Update slack invite link - take 2
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/20598 ) Change subject: [site] Update slack invite link - take 2 .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20598 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: gh-pages Gerrit-MessageType: comment Gerrit-Change-Id: Id3090bcb452c72873fedd32b2b3074ebd99b3f87 Gerrit-Change-Number: 20598 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Chennaka Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Comment-Date: Fri, 20 Oct 2023 00:04:41 + Gerrit-HasComments: No
[kudu-CR] refactor: move SetEncryptionFlags to constructors
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20592 to look at the new patch set (#2). Change subject: refactor: move SetEncryptionFlags to constructors .. refactor: move SetEncryptionFlags to constructors SetEncryptionFlags() is a test utility function used to enable or disable encryption in unit tests. In some parameterized tests, SetEncryptionFlags() calls are distributed in many test cases, there is a risk that missing to call it if add new test cases. This patch moves the function call to the constructors, also changes TEST_F to TEST_P for test LogBlockManagerTest.TestContainerBlockLimitingByMetadataSizeWithCompaction Change-Id: I67dca5a45f78de87309d0434c3b003a83f28e1a6 --- M src/kudu/consensus/consensus_meta-test.cc M src/kudu/fs/log_block_manager-test.cc M src/kudu/util/file_cache-test.cc 3 files changed, 42 insertions(+), 70 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/20592/2 -- To view, visit http://gerrit.cloudera.org:8080/20592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I67dca5a45f78de87309d0434c3b003a83f28e1a6 Gerrit-Change-Number: 20592 Gerrit-PatchSet: 2 Gerrit-Owner: Yingchun Lai Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai
[kudu-CR] refactor: move SetEncryptionFlags to constructors
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/20592 ) Change subject: refactor: move SetEncryptionFlags to constructors .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/20592/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20592/1//COMMIT_MSG@9 PS1, Line 9: util > nit: utility Done http://gerrit.cloudera.org:8080/#/c/20592/1//COMMIT_MSG@11 PS1, Line 11: parameterize > nit: parameterized Done -- To view, visit http://gerrit.cloudera.org:8080/20592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67dca5a45f78de87309d0434c3b003a83f28e1a6 Gerrit-Change-Number: 20592 Gerrit-PatchSet: 2 Gerrit-Owner: Yingchun Lai Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Fri, 20 Oct 2023 02:26:51 + Gerrit-HasComments: Yes
[kudu-CR] refactor: move SetEncryptionFlags to constructors
Yingchun Lai has removed a vote on this change. Change subject: refactor: move SetEncryptionFlags to constructors .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/20592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I67dca5a45f78de87309d0434c3b003a83f28e1a6 Gerrit-Change-Number: 20592 Gerrit-PatchSet: 2 Gerrit-Owner: Yingchun Lai Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai
[kudu-CR] refactor: move SetEncryptionFlags to constructors
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/20592 ) Change subject: refactor: move SetEncryptionFlags to constructors .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67dca5a45f78de87309d0434c3b003a83f28e1a6 Gerrit-Change-Number: 20592 Gerrit-PatchSet: 2 Gerrit-Owner: Yingchun Lai Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Fri, 20 Oct 2023 03:34:37 + Gerrit-HasComments: No
[kudu-CR] WIP Make --encrypt data at rest configurable for kudu-tool-test
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20597 to look at the new patch set (#2). Change subject: WIP Make --encrypt_data_at_rest configurable for kudu-tool-test .. WIP Make --encrypt_data_at_rest configurable for kudu-tool-test Change-Id: I5cba40850d2d2ef884f45e252b8117ae12b8f9d9 --- M src/kudu/tools/kudu-tool-test.cc 1 file changed, 188 insertions(+), 209 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/97/20597/2 -- To view, visit http://gerrit.cloudera.org:8080/20597 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5cba40850d2d2ef884f45e252b8117ae12b8f9d9 Gerrit-Change-Number: 20597 Gerrit-PatchSet: 2 Gerrit-Owner: Yingchun Lai Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3483 Fix flushing data in batch when table schema changed
Yingchun Lai has removed a vote on this change. Change subject: KUDU-3483 Fix flushing data in batch when table schema changed .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/19949 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ie6501962b32814d121f180b2942999c402d927db Gerrit-Change-Number: 19949 Gerrit-PatchSet: 20 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: KeDeng Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Yingchun Lai Gerrit-Reviewer: Yuqi Du
[kudu-CR] KUDU-3483 Fix flushing data in batch when table schema changed
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/19949 ) Change subject: KUDU-3483 Fix flushing data in batch when table schema changed .. Patch Set 20: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19949 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6501962b32814d121f180b2942999c402d927db Gerrit-Change-Number: 19949 Gerrit-PatchSet: 20 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: KeDeng Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Yingchun Lai Gerrit-Reviewer: Yuqi Du Gerrit-Comment-Date: Fri, 20 Oct 2023 06:16:51 + Gerrit-HasComments: No
[kudu-CR] KUDU-3407 Avoid unchecked scheduling of flush operations.
Hello Tidy Bot, Alexey Serbin, Ashwani Raina, Kudu Jenkins, Abhishek Chennaka, Wang Xixu, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20166 to look at the new patch set (#32). Change subject: KUDU-3407 Avoid unchecked scheduling of flush operations. .. KUDU-3407 Avoid unchecked scheduling of flush operations. In some clusters, the memory usages of tservers might be 60% ~ 80% for a long time. During this time the maintenance manager will not run any operation other than wal gc and MRS/DRS flushes, which will make the performance of tservers worse and worse and eventually break due to OOM. This patch add an argument to give a chance to do other operations while server is under memory pressure. This mechanism works when the memory usage is between memory_pressure_percentage and memory_limit_soft_percentage. Higher the memory usage is, higher the probability to flush MRS/DMS. e.g. memory_pressure_percentage = 60% memory_limit_soft_percentage = 80% The probability of not flushing MRS/DMS is the value of run_non_memory_ops_prob. As the memory increases, it gradually decreases to 0, when thememory usage is 80%. Change-Id: Idc2fd3a850cf99d54ef2980211b712468440ed80 --- M src/kudu/util/maintenance_manager-test.cc M src/kudu/util/maintenance_manager.cc M src/kudu/util/maintenance_manager.h 3 files changed, 200 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/20166/32 -- To view, visit http://gerrit.cloudera.org:8080/20166 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idc2fd3a850cf99d54ef2980211b712468440ed80 Gerrit-Change-Number: 20166 Gerrit-PatchSet: 32 Gerrit-Owner: Song Jiacheng Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Song Jiacheng Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com>
[kudu-CR] KUDU-3407 Avoid unchecked scheduling of flush operations.
Song Jiacheng has posted comments on this change. (
http://gerrit.cloudera.org:8080/20166 )
Change subject: KUDU-3407 Avoid unchecked scheduling of flush operations.
..
Patch Set 31:
(16 comments)
Thank you for the review and your attention for such a long time.
http://gerrit.cloudera.org:8080/#/c/20166/29//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20166/29//COMMIT_MSG@11
PS29, Line 11: wal
> nit: WAL
Done
http://gerrit.cloudera.org:8080/#/c/20166/29//COMMIT_MSG@12
PS29, Line 12: eventually break
: due to OOM
> I'm curious what do you think was the actual reason behind that OOM conditi
Yes, normally tablet servers reject write requests if their memory usage reach
80%, but sometimes the usage might be higher than 100% and got killed by
system(The system memory runs out).
I think the reason is that the running memory is getting higher because of the
worse performance. I could always find that the running memory would be high if
the cluster is lack of CPU or I/O. I think their root cause are the
same--read/write operation takes longer.
http://gerrit.cloudera.org:8080/#/c/20166/29//COMMIT_MSG@16
PS29, Line 16: server is under memory pressure
> By your experience with applying this functionality in a real cluster, woul
Not really actually, this patch has been applied in over 100 clusters for about
a year and the value run_non_memory_ops_prob is set to 0.2, I did not find any
OOM because of this.
The probability of running compactions is decreasing as the memory usage
getting higher. So actually the behaviors while the memory usage is higher than
80% are the same. And with better performance, read/write operation will not
take too long even if under memory pressure.
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc
File src/kudu/util/maintenance_manager-test.cc:
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@314
PS29, Line 314:
> nit: drop 'that'
Done
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@316
PS29, Line 316: y times the operat
> nit: usually it's called queue time or something, meaning how much time an
Done
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@318
PS29, Line 318: e oper
> nit: maybe, name this 'run_count_'?
Done.
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@389
PS29, Line 389:
> I guess it makes sense to rename this field since its semantics is differen
Done
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@982
PS29, Line 982: tMaintenanceOp op1("perf_op", MaintenanceOp::HIGH_IO_USAGE)
> What is the probability of actually choosing a memory flushing op here? In
This test does not make too much sense for now. The test below is actually
covering it, so delete it.
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@985
PS29, Line 985: p_time(MonoDelta::FromMilliseconds(5));
> nit for here and elsewhere for ASSERT_EQ(): the expected value comes first
Done
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@998
PS29, Line 998: ister_self(true);
> Even with probabilistic behavior, isn't it always possible at least to chec
Done
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@999
PS29, Line 999:
> Given this test scenario runs way over 3 seconds, please add SKIP_IF_SLOW_
Done
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@1006
PS29, Line 1006: FLAGS_enable_maintenance_manager = true;
: // Wait for the memory_op to run over 1000 times and then
check the running times
: // of other operations.
> Why to output this if all these variables aren't even changing at this poin
I thought the test is provided to the users to set different parameters and run
multi times, done.
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@1035
PS29, Line 1035: O) << Substitute("
> What is 'MaintenanceMgr num'?
It is the thread name of maintenance manager. I will make it clear.
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@1036
PS29, Line 1036:
> Wrap this with NO_FATALS() ?
Done
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@1044
PS29, Line 1044:
> Why to wait so long? Would be great to comment on this. Alternatively, ma
Done
http://gerrit.cloudera.org:8080/#/c/20166/29/src/kudu/util/maintenance_manager-test.cc@1046
PS29, Line 1046:
:
> Is there a way to check for the number of operations still running or opera
50ms is enough, which is also accepted, I think..
--
To view, visit http://gerrit.cloudera.org:8080/
[kudu-CR] Minor refactors on kudu-tool-test
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20597 to look at the new patch set (#3). Change subject: Minor refactors on kudu-tool-test .. Minor refactors on kudu-tool-test This patch makes minor refactors on kudu-tool-test.cc, includes: - Passes --encrypt_data_at_rest and --enable_multi_tenancy to CLI tools automatically, to avoid passing these parameters every time. - Wrap some code blocks with parenthesis to make the test cases more readability. Change-Id: I5cba40850d2d2ef884f45e252b8117ae12b8f9d9 --- M src/kudu/tools/kudu-tool-test.cc 1 file changed, 188 insertions(+), 205 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/97/20597/3 -- To view, visit http://gerrit.cloudera.org:8080/20597 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5cba40850d2d2ef884f45e252b8117ae12b8f9d9 Gerrit-Change-Number: 20597 Gerrit-PatchSet: 3 Gerrit-Owner: Yingchun Lai Gerrit-Reviewer: Kudu Jenkins (120)
