[kudu-CR] [KUDU-3447] Limit tablets copying speed
Hello Ashwani Raina, Yingchun Lai, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19479 to look at the new patch set (#8). Change subject: [KUDU-3447] Limit tablets copying speed .. [KUDU-3447] Limit tablets copying speed Copying tablets from one cluster to another is a high resource consumed operation using the command : kudu local_replica copy_from_remote. If the data size is very large, the copying process will last for a long time. Other service maybe get impacted and become unavailable because of the tablets copying process cost too much disk and/or network bandwith. Therefore it is better to limit the tablets copying speed and make the system more stable. The goal is a trade-off the tablets copying speed and the resource consumption. As copy_from_remote is mainly downloading data from the remote cluster and writing the data to local file system, it is better to control the downloading speed to control the resource consumption. This patch use a throttler to limit tablet copying speed. Change-Id: I1f4834bfb0718a2b6b1d946975287a11f6be1fe3 --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_local_replica.cc M src/kudu/tserver/tablet_copy_client.cc M src/kudu/tserver/tablet_copy_client.h 4 files changed, 79 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/79/19479/8 -- To view, visit http://gerrit.cloudera.org:8080/19479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1f4834bfb0718a2b6b1d946975287a11f6be1fe3 Gerrit-Change-Number: 19479 Gerrit-PatchSet: 8 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yingchun Lai
[kudu-CR] [KUDU-3447] Limit tablets copying speed
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19479 ) Change subject: [KUDU-3447] Limit tablets copying speed .. Patch Set 7: (4 comments) http://gerrit.cloudera.org:8080/#/c/19479/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19479/5//COMMIT_MSG@9 PS5, Line 9: one cluster to another is a high reso > nit: one cluster to another Done http://gerrit.cloudera.org:8080/#/c/19479/5//COMMIT_MSG@13 PS5, Line 13: get impacted and become unavailable > It is necessary to mention the reason, I guess it's because of the tablets Yes, you are right. http://gerrit.cloudera.org:8080/#/c/19479/5//COMMIT_MSG@15 PS5, Line 15: limit the > How about: "a tradeoff of" Done http://gerrit.cloudera.org:8080/#/c/19479/5/thirdparty/build-definitions.sh File thirdparty/build-definitions.sh: http://gerrit.cloudera.org:8080/#/c/19479/5/thirdparty/build-definitions.sh@1248 PS5, Line 1248: > Have you ever try class Throttler in src/kudu/util/throttler.h, does it mee Throttler is a simple rate limiter. To avoid dependence on the third party library, here use throttler as the rate limiter. -- To view, visit http://gerrit.cloudera.org:8080/19479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1f4834bfb0718a2b6b1d946975287a11f6be1fe3 Gerrit-Change-Number: 19479 Gerrit-PatchSet: 7 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yingchun Lai Gerrit-Comment-Date: Mon, 18 Sep 2023 04:36:30 + Gerrit-HasComments: Yes
[kudu-CR] [KUDU-3447] Limit tablets copying speed
Hello Ashwani Raina, Yingchun Lai, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19479 to look at the new patch set (#7). Change subject: [KUDU-3447] Limit tablets copying speed .. [KUDU-3447] Limit tablets copying speed Copying tablets from one cluster to another is a high resource consumed operation using the command : kudu local_replica copy_from_remote. If the data size is very large, the copying process will last for a long time. Other service maybe get impacted and become unavailable because of the tablets copying process cost too much disk and/or network bandwith. Therefore it is better to limit the tablets copying speed and make the system more stable. The goal is a trade-off the tablets copying speed and the resource consumption. As copy_from_remote is mainly downloading data from the remote cluster and writing the data to local file system, it is better to control the downloading speed to control the resource consumption. This patch use a throttler to limit tablet copying speed. Change-Id: I1f4834bfb0718a2b6b1d946975287a11f6be1fe3 --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_local_replica.cc M src/kudu/tserver/tablet_copy_client.cc M src/kudu/tserver/tablet_copy_client.h 4 files changed, 79 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/79/19479/7 -- To view, visit http://gerrit.cloudera.org:8080/19479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1f4834bfb0718a2b6b1d946975287a11f6be1fe3 Gerrit-Change-Number: 19479 Gerrit-PatchSet: 7 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yingchun Lai
[kudu-CR] [KUDU-3447] Limit tablets copying speed
Hello Ashwani Raina, Yingchun Lai, Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19479 to look at the new patch set (#6). Change subject: [KUDU-3447] Limit tablets copying speed .. [KUDU-3447] Limit tablets copying speed Copying tablets from one cluster to another is a high resource consumed operation using the command : kudu local_replica copy_from_remote. If the data size is very large, the copying process will last for a long time. Other service maybe get impacted and become unavailable because of the tablets copying process cost too much disk and/or network bandwith. Therefore it is better to limit the tablets copying speed and make the system more stable. The goal is a trade-off the tablets copying speed and the resource consumption. As copy_from_remote is mainly downloading data from the remote cluster and writing the data to local file system, it is better to control the downloading speed to control the resource consumption. This patch use a throttler to limit tablet copying speed. Change-Id: I1f4834bfb0718a2b6b1d946975287a11f6be1fe3 --- M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_local_replica.cc M src/kudu/tserver/tablet_copy_client.cc M src/kudu/tserver/tablet_copy_client.h M thirdparty/build-definitions.sh 5 files changed, 82 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/79/19479/6 -- To view, visit http://gerrit.cloudera.org:8080/19479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1f4834bfb0718a2b6b1d946975287a11f6be1fe3 Gerrit-Change-Number: 19479 Gerrit-PatchSet: 6 Gerrit-Owner: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Ashwani Raina Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Wang Xixu <1450306...@qq.com> Gerrit-Reviewer: Yingchun Lai
