Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
dataroaring merged PR #50062: URL: https://github.com/apache/doris/pull/50062 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
doris-robot commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2858285970 TeamCity cloud ut coverage result: Function Coverage: 83.18% (1103/1326) Line Coverage: 66.09% (18561/28084) Region Coverage: 65.65% (9239/14073) Branch Coverage: 55.42% (4966/8960) Coverage Report: http://coverage.selectdb-in.cc/coverage/b0efb3759c9a18954b26a621ccfedbcac00cc063_b0efb3759c9a18954b26a621ccfedbcac00cc063_cloud/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
github-actions[bot] commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2858282958 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
github-actions[bot] commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2858282862 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
hust-hhb commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2858245284 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
doris-robot commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2857155493 TeamCity cloud ut coverage result: Function Coverage: 83.09% (1101/1325) Line Coverage: 65.98% (18484/28016) Region Coverage: 65.47% (9170/14006) Branch Coverage: 55.38% (4932/8906) Coverage Report: http://coverage.selectdb-in.cc/coverage/161365b774219eb937a33f5f08dd5d923994322d_161365b774219eb937a33f5f08dd5d923994322d_cloud/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
hust-hhb commented on code in PR #50062:
URL: https://github.com/apache/doris/pull/50062#discussion_r2076834388
##
cloud/src/recycler/checker.cpp:
##
@@ -1189,4 +1195,75 @@ int
InstanceChecker::do_delete_bitmap_storage_optimize_check() {
return (failed_tablets_num > 0) ? 1 : 0;
}
+int InstanceChecker::do_mow_compaction_key_check() {
+std::unique_ptr it;
+std::string begin = mow_tablet_compaction_key({instance_id_, 0, 0});
+std::string end = mow_tablet_compaction_key({instance_id_, INT64_MAX, 0});
+MowTabletCompactionPB mow_tablet_compaction;
+do {
+std::unique_ptr txn;
+TxnErrorCode err = txn_kv_->create_txn(&txn);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to create txn";
+return -1;
+}
+err = txn->get(begin, end, &it);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to get mow tablet compaction key, err=" <<
err;
+return -1;
+}
+int64_t now = duration_cast(
+
std::chrono::system_clock::now().time_since_epoch())
+ .count();
+while (it->has_next() && !stopped()) {
+auto [k, v] = it->next();
+std::string_view k1 = k;
+k1.remove_prefix(1);
+std::vector, int,
int>> out;
+decode_key(&k1, &out);
+// 0x01 "meta" ${instance_id} "mow_tablet_comp" ${table_id}
${initiator}
+auto table_id = std::get(std::get<0>(out[3]));
+auto initiator = std::get(std::get<0>(out[4]));
+if (!mow_tablet_compaction.ParseFromArray(v.data(), v.size()))
[[unlikely]] {
+LOG(WARNING) << "failed to parse MowTabletCompactionPB";
+return -1;
+}
+int64_t expiration = mow_tablet_compaction.expiration();
+//check compaction key failed should meet both following two
condition:
+//1.compaction key is expired
+//2.table lock key is not found or key is not expired
+if (expiration < now -
config::compaction_key_check_expiration_diff_seconds) {
+std::string lock_key =
+meta_delete_bitmap_update_lock_key({instance_id_,
table_id, -1});
+std::string lock_val;
+err = txn->get(lock_key, &lock_val);
+std::string reason = "";
+if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+reason = "table lock key not found";
+
+} else {
+DeleteBitmapUpdateLockPB lock_info;
+if (!lock_info.ParseFromString(lock_val)) [[unlikely]] {
+LOG(WARNING) << "failed to parse
DeleteBitmapUpdateLockPB";
+return -1;
+}
+if (lock_info.expiration() > now) {
Review Comment:
@mymeiyi already change it,take a look plz.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
hust-hhb commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2857138116 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
doris-robot commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2856976687 TeamCity cloud ut coverage result: Function Coverage: 83.09% (1101/1325) Line Coverage: 65.98% (18483/28015) Region Coverage: 65.39% (9157/14004) Branch Coverage: 55.35% (4928/8904) Coverage Report: http://coverage.selectdb-in.cc/coverage/c827d6cca31014aea06cfffb1ffb9bfa43a05139_c827d6cca31014aea06cfffb1ffb9bfa43a05139_cloud/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
hust-hhb commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2856956108 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
hust-hhb commented on code in PR #50062:
URL: https://github.com/apache/doris/pull/50062#discussion_r2076731762
##
cloud/src/recycler/checker.cpp:
##
@@ -1189,4 +1195,75 @@ int
InstanceChecker::do_delete_bitmap_storage_optimize_check() {
return (failed_tablets_num > 0) ? 1 : 0;
}
+int InstanceChecker::do_mow_compaction_key_check() {
+std::unique_ptr it;
+std::string begin = mow_tablet_compaction_key({instance_id_, 0, 0});
+std::string end = mow_tablet_compaction_key({instance_id_, INT64_MAX, 0});
+MowTabletCompactionPB mow_tablet_compaction;
+do {
+std::unique_ptr txn;
+TxnErrorCode err = txn_kv_->create_txn(&txn);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to create txn";
+return -1;
+}
+err = txn->get(begin, end, &it);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to get mow tablet compaction key, err=" <<
err;
+return -1;
+}
+int64_t now = duration_cast(
+
std::chrono::system_clock::now().time_since_epoch())
+ .count();
+while (it->has_next() && !stopped()) {
+auto [k, v] = it->next();
+std::string_view k1 = k;
+k1.remove_prefix(1);
+std::vector, int,
int>> out;
+decode_key(&k1, &out);
+// 0x01 "meta" ${instance_id} "mow_tablet_comp" ${table_id}
${initiator}
+auto table_id = std::get(std::get<0>(out[3]));
+auto initiator = std::get(std::get<0>(out[4]));
+if (!mow_tablet_compaction.ParseFromArray(v.data(), v.size()))
[[unlikely]] {
+LOG(WARNING) << "failed to parse MowTabletCompactionPB";
+return -1;
+}
+int64_t expiration = mow_tablet_compaction.expiration();
+//check compaction key failed should meet both following two
condition:
+//1.compaction key is expired
+//2.table lock key is not found or key is not expired
+if (expiration < now -
config::compaction_key_check_expiration_diff_seconds) {
+std::string lock_key =
+meta_delete_bitmap_update_lock_key({instance_id_,
table_id, -1});
+std::string lock_val;
+err = txn->get(lock_key, &lock_val);
+std::string reason = "";
+if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+reason = "table lock key not found";
+
+} else {
+DeleteBitmapUpdateLockPB lock_info;
+if (!lock_info.ParseFromString(lock_val)) [[unlikely]] {
+LOG(WARNING) << "failed to parse
DeleteBitmapUpdateLockPB";
+return -1;
+}
+if (lock_info.expiration() > now) {
Review Comment:
升级过程中出现compaction key残留（compaction在新ms请求锁，去旧ms释放锁失败了，导致compaction key残留）
接下来会出现几个可能的场景：
a.导入抢到锁，抢锁过程中，会检查并删除过期的compaction key，锁没过期
b.导入抢到锁，抢锁过程中，会检查并删除过期的compaction key，执行完导入，并释放锁，此时锁不存在
c.导入抢锁释放锁，lock key不存在，compaction抢到锁，抢锁过程中会检查并删除过期的compaction key，锁没过期
d.compaction持续抢到锁，但是锁的过期时间已经不会更新（假设所有ms都走新路径），这个时候compaction key依然残留
综上，a，b，c这三种场景，应该会删掉compaction key，如果依然存在过期的compaction key，说明删除没work
所以checker这里，只要出现锁没过期或者锁不存在的场景，但是又有过期的compaction key，就说明删除执行的有问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
mymeiyi commented on code in PR #50062: URL: https://github.com/apache/doris/pull/50062#discussion_r2075455114 ## cloud/src/common/config.h: ## @@ -97,6 +97,9 @@ CONF_mInt64(recycle_task_threshold_seconds, "10800"); // 3h // **just for TEST** CONF_Bool(force_immediate_recycle, "false"); +CONF_mBool(enable_compaction_key_check, "false"); Review Comment: enable_mow_compaction_key_check is better -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
mymeiyi commented on code in PR #50062:
URL: https://github.com/apache/doris/pull/50062#discussion_r2075454048
##
cloud/src/recycler/checker.cpp:
##
@@ -1189,4 +1195,75 @@ int
InstanceChecker::do_delete_bitmap_storage_optimize_check() {
return (failed_tablets_num > 0) ? 1 : 0;
}
+int InstanceChecker::do_mow_compaction_key_check() {
+std::unique_ptr it;
+std::string begin = mow_tablet_compaction_key({instance_id_, 0, 0});
+std::string end = mow_tablet_compaction_key({instance_id_, INT64_MAX, 0});
+MowTabletCompactionPB mow_tablet_compaction;
+do {
+std::unique_ptr txn;
+TxnErrorCode err = txn_kv_->create_txn(&txn);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to create txn";
+return -1;
+}
+err = txn->get(begin, end, &it);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to get mow tablet compaction key, err=" <<
err;
+return -1;
+}
+int64_t now = duration_cast(
+
std::chrono::system_clock::now().time_since_epoch())
+ .count();
+while (it->has_next() && !stopped()) {
+auto [k, v] = it->next();
+std::string_view k1 = k;
+k1.remove_prefix(1);
+std::vector, int,
int>> out;
+decode_key(&k1, &out);
+// 0x01 "meta" ${instance_id} "mow_tablet_comp" ${table_id}
${initiator}
+auto table_id = std::get(std::get<0>(out[3]));
+auto initiator = std::get(std::get<0>(out[4]));
+if (!mow_tablet_compaction.ParseFromArray(v.data(), v.size()))
[[unlikely]] {
+LOG(WARNING) << "failed to parse MowTabletCompactionPB";
+return -1;
+}
+int64_t expiration = mow_tablet_compaction.expiration();
+//check compaction key failed should meet both following two
condition:
+//1.compaction key is expired
+//2.table lock key is not found or key is not expired
+if (expiration < now -
config::compaction_key_check_expiration_diff_seconds) {
+std::string lock_key =
+meta_delete_bitmap_update_lock_key({instance_id_,
table_id, -1});
+std::string lock_val;
+err = txn->get(lock_key, &lock_val);
+std::string reason = "";
+if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
+reason = "table lock key not found";
+
+} else {
+DeleteBitmapUpdateLockPB lock_info;
+if (!lock_info.ParseFromString(lock_val)) [[unlikely]] {
+LOG(WARNING) << "failed to parse
DeleteBitmapUpdateLockPB";
+return -1;
+}
+if (lock_info.expiration() > now) {
Review Comment:
should check lock_id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
doris-robot commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2853866938 TeamCity cloud ut coverage result: Function Coverage: 83.09% (1101/1325) Line Coverage: 65.94% (18472/28015) Region Coverage: 65.35% (9152/14004) Branch Coverage: 55.28% (4922/8904) Coverage Report: http://coverage.selectdb-in.cc/coverage/ea0e80bad07c576966408f1dbb50620421ed7868_ea0e80bad07c576966408f1dbb50620421ed7868_cloud/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
hust-hhb commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2853821647 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
zhannngchen commented on code in PR #50062:
URL: https://github.com/apache/doris/pull/50062#discussion_r2055594217
##
cloud/src/recycler/checker.cpp:
##
@@ -1189,4 +1195,52 @@ int
InstanceChecker::do_delete_bitmap_storage_optimize_check() {
return (failed_tablets_num > 0) ? 1 : 0;
}
+int InstanceChecker::do_compaction_key_check() {
+std::unique_ptr it;
+std::string begin = mow_tablet_compaction_key({instance_id_, 0, 0});
+std::string end = mow_tablet_compaction_key({instance_id_, INT64_MAX, 0});
+MowTabletCompactionPB mow_tablet_compaction;
+do {
+std::unique_ptr txn;
+TxnErrorCode err = txn_kv_->create_txn(&txn);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to create txn";
+return -1;
+}
+err = txn->get(begin, end, &it);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to get mow tablet compaction key, err=" <<
err;
+return -1;
+}
+int64_t now = duration_cast(
+
std::chrono::system_clock::now().time_since_epoch())
+ .count();
+while (it->has_next() && !stopped()) {
+auto [k, v] = it->next();
+std::string_view k1 = k;
+k1.remove_prefix(1);
+std::vector, int,
int>> out;
+decode_key(&k1, &out);
+// 0x01 "meta" ${instance_id} "mow_tablet_comp" ${table_id}
${initiator}
+auto table_id = std::get(std::get<0>(out[3]));
+auto initiator = std::get(std::get<0>(out[4]));
+LOG(INFO) << "table_id=" << table_id << ",initiator=" << initiator;
+if (!mow_tablet_compaction.ParseFromArray(v.data(), v.size()))
[[unlikely]] {
+LOG(WARNING) << "failed to parse MowTabletCompactionPB";
+return -1;
+}
+int64_t expiration = mow_tablet_compaction.expiration();
+if (expiration < now -
config::compaction_key_check_expiration_diff_seconds) {
+LOG(WARNING) << fmt::format(
+"[compaction key check fails] compaction key check
fail for "
+"instance_id={}, tablet_id={}, initiator={},
expiration={},now={} ",
Review Comment:
should be table_id rather than tablet_id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
zhannngchen commented on code in PR #50062:
URL: https://github.com/apache/doris/pull/50062#discussion_r2055466432
##
cloud/src/recycler/checker.h:
##
@@ -103,6 +103,8 @@ class InstanceChecker {
// which will not be cleared.
int do_delete_bitmap_storage_optimize_check();
+int do_compaction_key_check();
Review Comment:
```suggestion
int do_mow_compaction_key_check();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
zhannngchen commented on code in PR #50062:
URL: https://github.com/apache/doris/pull/50062#discussion_r2055482926
##
cloud/test/recycler_test.cpp:
##
@@ -3105,6 +3105,75 @@ TEST(CheckerTest,
delete_bitmap_storage_optimize_check_abnormal) {
ASSERT_EQ(expected_abnormal_rowsets, real_abnormal_rowsets);
}
+std::unique_ptr get_meta_service() {
+int ret = 0;
+// MemKv
+auto txn_kv =
std::dynamic_pointer_cast(std::make_shared());
+if (txn_kv != nullptr) {
+ret = txn_kv->init();
+[&] { ASSERT_EQ(ret, 0); }();
+}
+[&] { ASSERT_NE(txn_kv.get(), nullptr); }();
+
+std::unique_ptr txn;
+EXPECT_EQ(txn_kv->create_txn(&txn), TxnErrorCode::TXN_OK);
+txn->remove("\x00", "\xfe"); // This is dangerous if the fdb is not
correctly set
+EXPECT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
+
+auto rs = std::make_shared(txn_kv);
+auto rl = std::make_shared();
+auto meta_service = std::make_unique(txn_kv, rs, rl);
+return std::make_unique(std::move(meta_service));
+}
+
+MetaServiceCode get_delete_bitmap_lock(MetaServiceProxy* meta_service, int64_t
table_id,
+ int64_t lock_id, int64_t initiator,
int64_t expiration = 5) {
+brpc::Controller cntl;
+GetDeleteBitmapUpdateLockRequest req;
+GetDeleteBitmapUpdateLockResponse res;
+req.set_cloud_unique_id("test_cloud_unique_id");
+req.set_table_id(table_id);
+req.set_expiration(expiration);
+req.set_lock_id(lock_id);
+req.set_initiator(initiator);
+meta_service->get_delete_bitmap_update_lock(
+reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req,
&res, nullptr);
+return res.status().code();
+}
+
+TEST(CheckerTest, check_compaction_key) {
+config::enable_compaction_key_check = true;
+config::compaction_key_check_expiration_diff_seconds = 0;
+std::string instance_id = "test_check_compaction_key";
+[[maybe_unused]] auto sp = SyncPoint::get_instance();
+std::unique_ptr> defer(
+(int*)0x01, [](int*) {
SyncPoint::get_instance()->clear_all_call_backs(); });
+sp->set_call_back("get_instance_id", [&](auto&& args) {
+auto* ret = try_any_cast_ret(args);
+ret->first = instance_id;
+ret->second = true;
+});
+sp->enable_processing();
+
+auto meta_service = get_meta_service();
+auto res_code = get_delete_bitmap_lock(meta_service.get(), 2, -1, 123);
+ASSERT_EQ(res_code, MetaServiceCode::OK);
+std::this_thread::sleep_for(std::chrono::seconds(6));
+res_code = get_delete_bitmap_lock(meta_service.get(), 2, 100, -1);
+ASSERT_EQ(res_code, MetaServiceCode::OK);
+
+auto txn_kv = std::make_shared();
+ASSERT_EQ(txn_kv->init(), 0);
+
+InstanceInfoPB instance;
+instance.set_instance_id(instance_id);
+auto obj_info = instance.add_obj_info();
+obj_info->set_id("1");
+InstanceChecker checker(meta_service->txn_kv(), instance_id);
+ASSERT_EQ(checker.init(instance), 0);
+ASSERT_EQ(checker.do_compaction_key_check(), 0);
Review Comment:
This case also needs to cover exceptional scenarios, you should construct a
situation where the compaction key leaks, ensuring the checker can detect it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
zhannngchen commented on code in PR #50062:
URL: https://github.com/apache/doris/pull/50062#discussion_r2055465281
##
cloud/src/recycler/checker.cpp:
##
@@ -1189,4 +1195,52 @@ int
InstanceChecker::do_delete_bitmap_storage_optimize_check() {
return (failed_tablets_num > 0) ? 1 : 0;
}
+int InstanceChecker::do_compaction_key_check() {
+std::unique_ptr it;
+std::string begin = mow_tablet_compaction_key({instance_id_, 0, 0});
+std::string end = mow_tablet_compaction_key({instance_id_, INT64_MAX, 0});
+MowTabletCompactionPB mow_tablet_compaction;
+do {
+std::unique_ptr txn;
+TxnErrorCode err = txn_kv_->create_txn(&txn);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to create txn";
+return -1;
+}
+err = txn->get(begin, end, &it);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to get mow tablet compaction key, err=" <<
err;
+return -1;
+}
+int64_t now = duration_cast(
+
std::chrono::system_clock::now().time_since_epoch())
+ .count();
+while (it->has_next() && !stopped()) {
+auto [k, v] = it->next();
+std::string_view k1 = k;
+k1.remove_prefix(1);
+std::vector, int,
int>> out;
+decode_key(&k1, &out);
+// 0x01 "meta" ${instance_id} "mow_tablet_comp" ${table_id}
${initiator}
+auto table_id = std::get(std::get<0>(out[3]));
+auto initiator = std::get(std::get<0>(out[4]));
+LOG(INFO) << "table_id=" << table_id << ",initiator=" << initiator;
+if (!mow_tablet_compaction.ParseFromArray(v.data(), v.size()))
[[unlikely]] {
+LOG(WARNING) << "failed to parse MowTabletCompactionPB";
+return -1;
+}
+int64_t expiration = mow_tablet_compaction.expiration();
+if (expiration < now -
config::compaction_key_check_expiration_diff_seconds) {
+LOG(WARNING) << fmt::format(
Review Comment:
it’s also necessary to check whether the corresponding compaction lock key
still exists. If it does, this situation might be expected and does not require
an warning log and error response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
hust-hhb commented on code in PR #50062:
URL: https://github.com/apache/doris/pull/50062#discussion_r2055452498
##
cloud/src/recycler/checker.cpp:
##
@@ -1189,4 +1195,52 @@ int
InstanceChecker::do_delete_bitmap_storage_optimize_check() {
return (failed_tablets_num > 0) ? 1 : 0;
}
+int InstanceChecker::do_compaction_key_check() {
+std::unique_ptr it;
+std::string begin = mow_tablet_compaction_key({instance_id_, 0, 0});
+std::string end = mow_tablet_compaction_key({instance_id_, INT64_MAX, 0});
+MowTabletCompactionPB mow_tablet_compaction;
+do {
+std::unique_ptr txn;
+TxnErrorCode err = txn_kv_->create_txn(&txn);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to create txn";
+return -1;
+}
+err = txn->get(begin, end, &it);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to get mow tablet compaction key, err=" <<
err;
+return -1;
+}
+int64_t now = duration_cast(
+
std::chrono::system_clock::now().time_since_epoch())
+ .count();
+while (it->has_next() && !stopped()) {
+auto [k, v] = it->next();
+std::string_view k1 = k;
+k1.remove_prefix(1);
+std::vector, int,
int>> out;
+decode_key(&k1, &out);
+// 0x01 "mow" ${instance_id} "mow_tablet_comp" ${table_id}
${initiator}
Review Comment:
@zhannngchen done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
zhannngchen commented on code in PR #50062:
URL: https://github.com/apache/doris/pull/50062#discussion_r2055434116
##
cloud/src/recycler/checker.cpp:
##
@@ -1189,4 +1195,52 @@ int
InstanceChecker::do_delete_bitmap_storage_optimize_check() {
return (failed_tablets_num > 0) ? 1 : 0;
}
+int InstanceChecker::do_compaction_key_check() {
+std::unique_ptr it;
+std::string begin = mow_tablet_compaction_key({instance_id_, 0, 0});
+std::string end = mow_tablet_compaction_key({instance_id_, INT64_MAX, 0});
+MowTabletCompactionPB mow_tablet_compaction;
+do {
+std::unique_ptr txn;
+TxnErrorCode err = txn_kv_->create_txn(&txn);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to create txn";
+return -1;
+}
+err = txn->get(begin, end, &it);
+if (err != TxnErrorCode::TXN_OK) {
+LOG(WARNING) << "failed to get mow tablet compaction key, err=" <<
err;
+return -1;
+}
+int64_t now = duration_cast(
+
std::chrono::system_clock::now().time_since_epoch())
+ .count();
+while (it->has_next() && !stopped()) {
+auto [k, v] = it->next();
+std::string_view k1 = k;
+k1.remove_prefix(1);
+std::vector, int,
int>> out;
+decode_key(&k1, &out);
+// 0x01 "mow" ${instance_id} "mow_tablet_comp" ${table_id}
${initiator}
Review Comment:
update the comment, #50242 changed the key format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
doris-robot commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2804834143 TeamCity cloud ut coverage result: Function Coverage: 83.02% (1090/1313) Line Coverage: 66.01% (18188/27552) Region Coverage: 65.47% (8961/13687) Branch Coverage: 55.27% (4825/8730) Coverage Report: http://coverage.selectdb-in.cc/coverage/17cadc4f083fa77aed57ceb671a3bb6efc366254_17cadc4f083fa77aed57ceb671a3bb6efc366254_cloud/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
hust-hhb commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2804809729 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](cloud-mow)Add checker for compaction key [doris]
hello-stephen commented on PR #50062: URL: https://github.com/apache/doris/pull/50062#issuecomment-2804809568 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
