[kudu-CR] [Tool] Limit table copying speed
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 6: (9 comments) http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc File src/kudu/tools/kudu-tool-test.cc: http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6011 PS6, Line 6011: 10240 > Are you sure this is small enough to track copying of 2 rows, especiall Actually, the data size is 881500. Every batch size is about 5248, total 170 batches is readout. So, every 2 batches will be block as the table_copy_throttler_bytes_per_sec is 10240. That will block for at least 85 seconds. Please see my next patch: https://gerrit.cloudera.org/c/21609 http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6029 PS6, Line 6029: client->OpenTable(kNewTableName, &table); > Wrap this into ASSERT_OK()? Done http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6031 PS6, Line 6031: scanner.Open(); > Wrap this into ASSERT_OK()? Done http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6036 PS6, Line 6036: = > Shouldn't this be '+=' instead of '='? With the current code, the assertio Sorry, that is a mistake. http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6038 PS6, Line 6038: must > nit: must be Done http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.h File src/kudu/tools/table_scanner.h: http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.h@107 PS6, Line 107: std::shared_ptr > Why shared_ptr? There isn't any shared ownership for this field, this is so You are right. Thanks. http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc File src/kudu/tools/table_scanner.cc: http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@611 PS6, Line 611: copy > nit: copying Done http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@613 PS6, Line 613: SCOPED_LOG_SLOW_EXECUTION > What's the purpose of this warning? If the idea was to track cases when it Done http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@614 PS6, Line 614: 0 > Ah, this 0 is for op tokens, and the 'bytes' consumed is set as necessary, Done -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 6 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Wed, 24 Jul 2024 10:17:46 + Gerrit-HasComments: Yes
[kudu-CR] [Tool] Limit table copying speed
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc File src/kudu/tools/table_scanner.cc: http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@614 PS6, Line 614: 0 > Why 0? Isn't this a typo? Would be great to at least add a comment on thi Ah, this 0 is for op tokens, and the 'bytes' consumed is set as necessary, so this one should be good. Please ignore this particular comment. -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 6 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Fri, 19 Jul 2024 18:48:17 + Gerrit-HasComments: Yes
[kudu-CR] [Tool] Limit table copying speed
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 6: (9 comments) I'm not sure this works as expected in PS6. http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc File src/kudu/tools/kudu-tool-test.cc: http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6011 PS6, Line 6011: 10240 Are you sure this is small enough to track copying of 2 rows, especially of the data is compressed? http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6029 PS6, Line 6029: client->OpenTable(kNewTableName, &table); Wrap this into ASSERT_OK()? http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6031 PS6, Line 6031: scanner.Open(); Wrap this into ASSERT_OK()? http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6036 PS6, Line 6036: = Shouldn't this be '+=' instead of '='? With the current code, the assertion at 6039 doesn't make much sense if 'data_size' just tracks the size of the very last batch. http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6038 PS6, Line 6038: must nit: must be http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.h File src/kudu/tools/table_scanner.h: http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.h@107 PS6, Line 107: std::shared_ptr Why shared_ptr? There isn't any shared ownership for this field, this is solely own by TableScanner instance. Should this be std::unique_ptr instead? http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc File src/kudu/tools/table_scanner.cc: http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@611 PS6, Line 611: copy nit: copying http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@613 PS6, Line 613: SCOPED_LOG_SLOW_EXECUTION What's the purpose of this warning? If the idea was to track cases when it takes a long time to consume a throttler's token, semantically that seems closer to INFO, not WARNING since it's expected to throttle as configured. http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@614 PS6, Line 614: 0 Why 0? Isn't this a typo? Would be great to at least add a comment on this -- I'd expect this to be at least 1. -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 6 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Fri, 19 Jul 2024 18:18:27 + Gerrit-HasComments: Yes
[kudu-CR] [Tool] Limit table copying speed
Yifan Zhang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. [Tool] Limit table copying speed Migrating data to another Kudu cluster using the 'kudu table copy' CLI command when the data is very large may cause memory and/or network bandwidth perssure. To reduce the effect on other services, it is better to limit the copying speed. This patch introduces two parameter: --table_copy_throttler_bytes_per_sec and --table_copy_throttler_burst_factor to limit the table copying speed. Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Reviewed-on: http://gerrit.cloudera.org:8080/21527 Reviewed-by: Zoltan Chovan Reviewed-by: Abhishek Chennaka Tested-by: Yifan Zhang Reviewed-by: Yifan Zhang --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/table_scanner.cc M src/kudu/tools/table_scanner.h M src/kudu/tools/tool_action_table.cc 4 files changed, 81 insertions(+), 0 deletions(-) Approvals: Zoltan Chovan: Looks good to me, but someone else must approve Abhishek Chennaka: Looks good to me, approved Yifan Zhang: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 6 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka
[kudu-CR] [Tool] Limit table copying speed
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 5 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Fri, 19 Jul 2024 12:00:01 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 5: Verified+1 Unrelated test failures. -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 5 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yifan Zhang Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Fri, 19 Jul 2024 11:59:52 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Yifan Zhang has removed a vote on this change. Change subject: [Tool] Limit table copying speed .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 5 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka
[kudu-CR] [Tool] Limit table copying speed
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 5 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Wed, 10 Jul 2024 17:32:51 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Zoltan Chovan has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 5 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Tue, 09 Jul 2024 09:02:39 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 5: Verified-1 Build Failed http://jenkins.kudu.apache.org/job/pre_commit/285/ : FAILURE -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 5 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Tue, 02 Jul 2024 07:40:05 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 5: Build Started http://jenkins.kudu.apache.org/job/pre_commit/285/ -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 5 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Tue, 02 Jul 2024 06:29:33 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Hello Marton Greber, Zoltan Chovan, Zoltan Martonka, Attila Bukor, Kudu Jenkins, Abhishek Chennaka, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21527 to look at the new patch set (#5). Change subject: [Tool] Limit table copying speed .. [Tool] Limit table copying speed Migrating data to another Kudu cluster using the 'kudu table copy' CLI command when the data is very large may cause memory and/or network bandwidth perssure. To reduce the effect on other services, it is better to limit the copying speed. This patch introduces two parameter: --table_copy_throttler_bytes_per_sec and --table_copy_throttler_burst_factor to limit the table copying speed. Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/table_scanner.cc M src/kudu/tools/table_scanner.h M src/kudu/tools/tool_action_table.cc 4 files changed, 81 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/5 -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 5 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 4: Verified-1 Build Failed http://jenkins.kudu.apache.org/job/pre_commit/284/ : FAILURE -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 4 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Tue, 02 Jul 2024 06:25:22 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Hello Marton Greber, Zoltan Chovan, Zoltan Martonka, Attila Bukor, Kudu Jenkins, Abhishek Chennaka, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21527 to look at the new patch set (#4). Change subject: [Tool] Limit table copying speed .. [Tool] Limit table copying speed Migrating data to another Kudu cluster using the 'kudu table copy' CLI command when the data is very large may cause memory and/or network bandwidth perssure. To reduce the effect on other services, it is better to limit the copying speed. This patch introduces two parameter: --table_copy_throttler_bytes_per_sec and --table_copy_throttler_burst_factor to limit the table copying speed. Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/table_scanner.cc M src/kudu/tools/table_scanner.h M src/kudu/tools/tool_action_table.cc 4 files changed, 79 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/4 -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 4 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 4: Build Started http://jenkins.kudu.apache.org/job/pre_commit/284/ -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 4 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Tue, 02 Jul 2024 06:14:03 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 3: Verified-1 Build Failed http://jenkins.kudu.apache.org/job/pre_commit/279/ : FAILURE -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 3 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Mon, 01 Jul 2024 03:08:02 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc File src/kudu/tools/table_scanner.cc: http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@146 PS2, Line 146: second" > nit: second Done http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@149 PS2, Line 149: multipli > nit: multiplied by Done http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@150 PS2, Line 150: base rate > nit: base rate (--table_copy_throttler_bytes_per_sec). Done -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 3 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Mon, 01 Jul 2024 02:57:56 + Gerrit-HasComments: Yes
[kudu-CR] [Tool] Limit table copying speed
Hello Marton Greber, Zoltan Chovan, Zoltan Martonka, Attila Bukor, Kudu Jenkins, Abhishek Chennaka, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21527 to look at the new patch set (#3). Change subject: [Tool] Limit table copying speed .. [Tool] Limit table copying speed Migrating data to another Kudu cluster using the 'kudu table copy' CLI command when the data is very large may cause memory and/or network bandwidth perssure. To reduce the effect on other services, it is better to limit the copying speed. This patch introduces two parameter: --table_copy_throttler_bytes_per_sec and --table_copy_throttler_burst_factor to limit the table copying speed. Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/table_scanner.cc M src/kudu/tools/table_scanner.h M src/kudu/tools/tool_action_table.cc 4 files changed, 79 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/3 -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 3 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 3: Build Started http://jenkins.kudu.apache.org/job/pre_commit/279/ -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 3 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Mon, 01 Jul 2024 02:57:49 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc File src/kudu/tools/table_scanner.cc: http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@146 PS2, Line 146: seconds nit: second http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@149 PS2, Line 149: multiply nit: multiplied by http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@150 PS2, Line 150: base rate nit: base rate (--table_copy_throttler_bytes_per_sec). -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 2 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Wed, 26 Jun 2024 23:47:45 + Gerrit-HasComments: Yes
[kudu-CR] [Tool] Limit table copying speed
Zoltan Chovan has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 2 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Reviewer: Zoltan Martonka Gerrit-Comment-Date: Fri, 21 Jun 2024 13:52:25 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 2: Verified-1 Build Failed http://jenkins.kudu.apache.org/job/pre_commit/265/ : FAILURE -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 2 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Comment-Date: Fri, 21 Jun 2024 03:36:12 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 2: (8 comments) http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@10 PS1, Line 10: se > nit: 'it' not needed Done http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@9 PS1, Line 9: using the 'kudu table copy' : CLI command when t > reword: using the 'kudu table copy' CLI command Done http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@10 PS1, Line 10: he d > nit: when Done http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@11 PS1, Line 11: network bandwidth perssure. To reduce the effect > memory and/or network bandwidth perssure Done http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@12 PS1, Line 12: s better to limit the c > effect on other services Done http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/kudu-tool-test.cc File src/kudu/tools/kudu-tool-test.cc: http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/kudu-tool-test.cc@5942 PS1, Line 5942: "--table_copy_throttler_burst_factor=100", > consider extracting this to it's own variable as it is used multiple times Done http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc File src/kudu/tools/table_scanner.cc: http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc@150 PS1, Line 150: maximum > nit:maximum Done http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc@150 PS1, Line 150: equal > nit:equal Done -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 2 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Zoltan Chovan Gerrit-Comment-Date: Fri, 21 Jun 2024 03:16:44 + Gerrit-HasComments: Yes
[kudu-CR] [Tool] Limit table copying speed
Hello Zoltan Chovan, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21527 to look at the new patch set (#2). Change subject: [Tool] Limit table copying speed .. [Tool] Limit table copying speed Migrating data to another Kudu cluster using the 'kudu table copy' CLI command when the data is very large may cause memory and/or network bandwidth perssure. To reduce the effect on other services, it is better to limit the copying speed. This patch introduces two parameter: --table_copy_throttler_bytes_per_sec and --table_copy_throttler_burst_factor to limit the table copying speed. Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/table_scanner.cc M src/kudu/tools/table_scanner.h M src/kudu/tools/tool_action_table.cc 4 files changed, 79 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/2 -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 2 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Chovan
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 2: Build Started http://jenkins.kudu.apache.org/job/pre_commit/265/ -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 2 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Chovan Gerrit-Comment-Date: Fri, 21 Jun 2024 03:16:39 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Zoltan Chovan has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 1: (8 comments) http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@10 PS1, Line 10: When nit: when http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@9 PS1, Line 9: using CLI command : 'kudu table copy' reword: using the 'kudu table copy' CLI command http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@10 PS1, Line 10: it nit: 'it' not needed http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@11 PS1, Line 11: memory pressure or run out the network bandwidth. memory and/or network bandwidth perssure http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@12 PS1, Line 12: affect on other service effect on other services http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/kudu-tool-test.cc File src/kudu/tools/kudu-tool-test.cc: http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/kudu-tool-test.cc@5942 PS1, Line 5942: cluster_->master()->bound_rpc_addr().ToString() consider extracting this to it's own variable as it is used multiple times in the test, e.g. string master_addr = cluster_->master()->bound_rpc_addr().ToString(); http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc File src/kudu/tools/table_scanner.cc: http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc@150 PS1, Line 150: maximun nit:maximum http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc@150 PS1, Line 150: equals nit:equal -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 1 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Chovan Gerrit-Comment-Date: Thu, 20 Jun 2024 09:56:04 + Gerrit-HasComments: Yes
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 1: Verified-1 Build Failed http://jenkins.kudu.apache.org/job/pre_commit/252/ : FAILURE -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 1 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Tue, 18 Jun 2024 12:08:41 + Gerrit-HasComments: No
[kudu-CR] [Tool] Limit table copying speed
Wang Xixu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21527 Change subject: [Tool] Limit table copying speed .. [Tool] Limit table copying speed Migrating data to another Kudu cluster using CLI command 'kudu table copy' When the data is very large, it may cause memory pressure or run out the network bandwidth. To reduce the affect on other service, it is better to limit the copying speed. This patch introduces two parameter: --table_copy_throttler_bytes_per_sec and --table_copy_throttler_burst_factor to limit the table copying speed. Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/table_scanner.cc M src/kudu/tools/table_scanner.h M src/kudu/tools/tool_action_table.cc 4 files changed, 78 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/1 -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 1 Gerrit-Owner: Wang Xixu <1450306...@qq.com>
[kudu-CR] [Tool] Limit table copying speed
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 ) Change subject: [Tool] Limit table copying speed .. Patch Set 1: Build Started http://jenkins.kudu.apache.org/job/pre_commit/252/ -- To view, visit http://gerrit.cloudera.org:8080/21527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3 Gerrit-Change-Number: 21527 Gerrit-PatchSet: 1 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Tue, 18 Jun 2024 09:01:45 + Gerrit-HasComments: No
