Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4964808042 only merge filter packed slice deletion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
liaoxin01 merged PR #63295: URL: https://github.com/apache/doris/pull/63295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
github-actions[bot] commented on code in PR #63295:
URL: https://github.com/apache/doris/pull/63295#discussion_r3353015646
##
cloud/test/recycler_test.cpp:
##
@@ -1376,6 +1400,89 @@ TEST(RecyclerTest, recycle_rowsets) {
check_delete_bitmap_file_size(accessor, tablet_id, 0);
}
+TEST(RecyclerTest, next_recycle_rowset_tablet_key_overwrites_existing_buffer) {
+std::string next_key = recycle_rowset_key({instance_id, 10002, "rowset"});
+ASSERT_EQ(InstanceRecycler::next_recycle_rowset_tablet_key(instance_id,
10002, &next_key), 0);
+
+std::string_view k1 = next_key;
+k1.remove_prefix(1);
+std::vector, int, int>> out;
+ASSERT_EQ(decode_key(&k1, &out), 0);
+EXPECT_EQ(std::get(std::get<0>(out[3])), 10003);
+EXPECT_TRUE(std::get(std::get<0>(out[4])).empty());
+}
+
+TEST(RecyclerTest,
recycle_rowsets_tablet_batch_limit_recycles_remaining_in_next_round) {
+config::retention_seconds = 0;
+auto txn_kv = std::make_shared();
+ASSERT_EQ(txn_kv->init(), 0);
+
+InstanceInfoPB instance;
+instance.set_instance_id(instance_id);
+auto obj_info = instance.add_obj_info();
+obj_info->set_id("recycle_rowsets_batch_limit");
+obj_info->set_ak(config::test_s3_ak);
+obj_info->set_sk(config::test_s3_sk);
+obj_info->set_endpoint(config::test_s3_endpoint);
+obj_info->set_region(config::test_s3_region);
+obj_info->set_bucket(config::test_s3_bucket);
+obj_info->set_prefix("recycle_rowsets_batch_limit");
+
+auto old_worker_pool_size = config::instance_recycler_worker_pool_size;
+auto old_max_rowsets_per_tablet =
config::recycle_rowsets_per_tablet_batch_size;
+auto old_enable_mark = config::enable_mark_delete_rowset_before_recycle;
+config::instance_recycler_worker_pool_size = 1;
+config::recycle_rowsets_per_tablet_batch_size = 3;
+config::enable_mark_delete_rowset_before_recycle = false;
+DORIS_CLOUD_DEFER {
+config::instance_recycler_worker_pool_size = old_worker_pool_size;
+config::recycle_rowsets_per_tablet_batch_size =
old_max_rowsets_per_tablet;
+config::enable_mark_delete_rowset_before_recycle = old_enable_mark;
+};
+
+InstanceRecycler recycler(txn_kv, instance, thread_group,
+ std::make_shared(txn_kv));
+ASSERT_EQ(recycler.init(), 0);
+auto accessor = recycler.accessor_map_.begin()->second;
+
+doris::TabletSchemaCloudPB schema;
+schema.set_schema_version(1);
+constexpr int64_t index_id = 10001;
+constexpr int64_t first_tablet_id = 10002;
+constexpr int64_t second_tablet_id = 10020;
+std::vector first_tablet_rowset_ids;
+for (int i = 0; i < 5; ++i) {
+auto rowset =
+create_rowset("recycle_rowsets_batch_limit", first_tablet_id,
index_id, 1, schema);
+first_tablet_rowset_ids.push_back(rowset.rowset_id_v2());
+ASSERT_EQ(create_recycle_rowset(txn_kv.get(), accessor.get(), rowset,
+RecycleRowsetPB::COMPACT, true),
+ 0);
+}
+for (int i = 0; i < 2; ++i) {
+auto rowset =
+create_rowset("recycle_rowsets_batch_limit", second_tablet_id,
index_id, 1, schema);
+ASSERT_EQ(create_recycle_rowset(txn_kv.get(), accessor.get(), rowset,
+RecycleRowsetPB::COMPACT, true),
+ 0);
Review Comment:
This test directly calls `InstanceRecycler::next_recycle_rowset_tablet_key`,
but that helper is declared in the private section of `InstanceRecycler` in
`recycler.h`. `recycler_test.cpp` includes `recycler/recycler.h` normally and
there is no `friend` or `#define private public`, so this line will fail to
compile with an access-control error. Please either keep the helper testable
through the public recycle path, make the helper accessible intentionally for
tests, or remove this direct unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4618338708 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
hello-stephen commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4615955596 # Cloud UT Coverage Report Increment line coverage `90.28% (223/247)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/39b290677871a9204ba1bbbfe7090e95587b3f14_39b290677871a9204ba1bbbfe7090e95587b3f14_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/39b290677871a9204ba1bbbfe7090e95587b3f14_39b290677871a9204ba1bbbfe7090e95587b3f14_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 78.39% (1915/2443) | | Line Coverage | 64.87% (34188/52701) | | Region Coverage | 65.28% (17575/26922) | | Branch Coverage | 53.98% (9335/17294) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4614782609 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
github-actions[bot] commented on code in PR #63295:
URL: https://github.com/apache/doris/pull/63295#discussion_r3317782929
##
cloud/src/recycler/recycler.cpp:
##
@@ -5152,18 +5208,123 @@ int InstanceRecycler::recycle_rowsets() {
LOG(WARNING) << "failed to delete recycle rowset kv,
instance_id=" << instance_id_;
return;
}
+
num_recycled.fetch_add(rowset_keys_to_delete.size(),
std::memory_order_relaxed);
});
+};
+
+auto submit_mark_abort_rowset_job = [&](std::vector
rowset_keys_to_mark,
+std::vector
rowset_keys_to_abort) {
+if (rowset_keys_to_mark.empty() && rowset_keys_to_abort.empty()) {
+return;
+}
+mark_abort_worker_pool->submit([&, rowset_keys_to_mark =
std::move(rowset_keys_to_mark),
+rowset_keys_to_abort =
+
std::move(rowset_keys_to_abort)]() mutable {
+auto start = steady_clock::now();
+DORIS_CLOUD_DEFER {
+auto cost = duration_cast(steady_clock::now() -
start).count();
+LOG(INFO) << "finish mark and abort rowset job, instance_id="
<< instance_id_ << ' '
+ << "cost_ms=" << cost << ' '
+ << "rowset_keys_to_mark.size()=" <<
rowset_keys_to_mark.size() << ' '
+ << "rowset_keys_to_abort.size()=" <<
rowset_keys_to_abort.size();
+};
+if (!rowset_keys_to_mark.empty() &&
+batch_mark_rowsets_as_recycled(txn_kv_.get(),
instance_id_,
+
rowset_keys_to_mark) != 0) {
+LOG(WARNING) << "failed to batch mark rowsets as recycled,
instance_id="
+ << instance_id_ << ' '
+ << "rowset_keys_to_mark.size()=" <<
rowset_keys_to_mark.size();
+return;
+}
+if (!rowset_keys_to_abort.empty() &&
+
batch_abort_txn_or_job_for_recycle(rowset_keys_to_abort, true)
!=
+0) {
+LOG(WARNING) << "failed to batch abort txn or job for related
rowset, "
+"instance_id="
+ << instance_id_ << ' '
+ << "rowset_keys_to_abort.size()=" <<
rowset_keys_to_abort.size();
+return;
+}
+});
Review Comment:
This flush gate only considers rowsets that have data, so metadata-only
entries in `rowset_keys_without_data` can grow without bound until the entire
scan finishes. For example, a backlog of expired empty rowsets (`num_segments()
== 0`) or old-version rowsets with empty `resource_id` keeps `rowsets.size() ==
0`, so every `loop_done()` during the scan returns here and no KV-removal jobs
are submitted until the final post-scan call. The old code swapped and
submitted `rowset_keys` on each scan batch, while this defeats the new
batching/memory-control intent and can retain all such keys for a large recycle
backlog. Please include `rowset_keys_without_data` in the flush threshold, or
flush it independently before returning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4563717919 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4561930698 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
github-actions[bot] commented on code in PR #63295:
URL: https://github.com/apache/doris/pull/63295#discussion_r3315965235
##
cloud/src/recycler/recycler.cpp:
##
@@ -5152,18 +5208,123 @@ int InstanceRecycler::recycle_rowsets() {
LOG(WARNING) << "failed to delete recycle rowset kv,
instance_id=" << instance_id_;
return;
}
+
num_recycled.fetch_add(rowset_keys_to_delete.size(),
std::memory_order_relaxed);
});
+};
+
+auto submit_mark_abort_rowset_job = [&](std::vector
rowset_keys_to_mark,
+std::vector
rowset_keys_to_abort) {
+if (rowset_keys_to_mark.empty() && rowset_keys_to_abort.empty()) {
+return;
+}
+mark_abort_worker_pool->submit([&, rowset_keys_to_mark =
std::move(rowset_keys_to_mark),
+rowset_keys_to_abort =
+
std::move(rowset_keys_to_abort)]() mutable {
+auto start = steady_clock::now();
+DORIS_CLOUD_DEFER {
+auto cost = duration_cast(steady_clock::now() -
start).count();
+LOG(INFO) << "finish mark and abort rowset job, instance_id="
<< instance_id_ << ' '
+ << "cost_ms=" << cost << ' '
+ << "rowset_keys_to_mark.size()=" <<
rowset_keys_to_mark.size() << ' '
+ << "rowset_keys_to_abort.size()=" <<
rowset_keys_to_abort.size();
+};
+if (!rowset_keys_to_mark.empty() &&
+batch_mark_rowsets_as_recycled(txn_kv_.get(),
instance_id_,
+
rowset_keys_to_mark) != 0) {
+LOG(WARNING) << "failed to batch mark rowsets as recycled,
instance_id="
+ << instance_id_ << ' '
+ << "rowset_keys_to_mark.size()=" <<
rowset_keys_to_mark.size();
+return;
+}
+if (!rowset_keys_to_abort.empty() &&
+
batch_abort_txn_or_job_for_recycle(rowset_keys_to_abort, true)
!=
+0) {
+LOG(WARNING) << "failed to batch abort txn or job for related
rowset, "
+"instance_id="
+ << instance_id_ << ' '
+ << "rowset_keys_to_abort.size()=" <<
rowset_keys_to_abort.size();
+return;
+}
+});
+};
+
+bool scan_finished = false;
+auto loop_done = [&]() -> int {
+std::vector mark_keys_to_process;
+std::vector abort_keys_to_process;
+mark_keys_to_process.swap(rowset_keys_to_mark_recycled);
+abort_keys_to_process.swap(rowset_keys_to_abort);
+submit_mark_abort_rowset_job(std::move(mark_keys_to_process),
+ std::move(abort_keys_to_process));
Review Comment:
This changes the ordering that protected PREPARE recycle rowsets.
`handle_rowset_kv` still queues keys in `rowset_keys_to_abort`, but for
`RecycleRowsetPB::PREPARE` it now calls `delete_rowset_data_by_prefix(...)`
immediately while the abort is only submitted here to `mark_abort_worker_pool`;
for non-PREPARE rows the delete jobs below can also run in `worker_pool` before
this separate pool finishes. The previous code ran mark, abort, prepare-task
collection, and deletion sequentially inside one worker job, so load rowsets
aborted their txn and compaction/schema-change rowsets aborted their job before
any physical files were removed. With this race, a failed abort can leave
metadata referring to rowset data that has already been deleted. Please keep
abort-before-delete in the same execution chain (or otherwise wait for the
abort/mark batch to finish successfully before submitting any delete job for
those keys).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
github-actions[bot] commented on code in PR #63295:
URL: https://github.com/apache/doris/pull/63295#discussion_r3315949501
##
cloud/src/recycler/recycler.cpp:
##
@@ -5067,78 +5172,29 @@ int InstanceRecycler::recycle_rowsets() {
if (rowset.type() == RecycleRowsetPB::PREPARE) {
// unable to calculate file path, can only be deleted by rowset id
prefix
num_prepare += 1;
-prepare_rowset_keys_to_delete.emplace_back(k);
+if (delete_rowset_data_by_prefix(std::string(k),
rowset_meta->resource_id(),
+ rowset_meta->tablet_id(),
Review Comment:
This deletes PREPARE rowset data before the related txn/job abort is
guaranteed to finish. Above, `rowset_keys_to_abort` is only queued, and
`loop_done()` now submits the abort work to `mark_abort_worker_pool`
independently from the delete work. For a prepared load rowset, this branch can
call `delete_rowset_data_by_prefix()` while the transaction is still
commit-able; if the commit wins that race, the committed rowset will point at
files the recycler just removed. The old code executed
`batch_abort_txn_or_job_for_recycle()` in the same worker before
collecting/deleting prepare rowsets, so please preserve that ordering, for
example by keeping prepare deletion in the same job after abort completes.
##
cloud/src/recycler/recycler.cpp:
##
@@ -4892,20 +4942,69 @@ int InstanceRecycler::recycle_rowsets() {
.tag("expired_rowset_meta_size", expired_rowset_size);
};
-std::vector rowset_keys;
+struct RecycleRowsetEntry {
+std::string key;
+doris::RowsetMetaCloudPB meta;
+};
+struct RecycleRowsetDeleteJob {
+std::vector keys;
+std::map rowsets;
+};
+// Store the scanned recycle key with rowset meta. The scanned key is the
actual KV key to delete.
+std::vector rowsets;
+int64_t current_tablet_id = -1;
+int64_t recycled_rowset_count_for_current_tablet = 0;
+bool current_tablet_skip_logged = false;
+std::string next_scan_begin;
+const int64_t rowset_batch_size_per_tablet =
+std::max(1, config::recycle_rowsets_per_tablet_batch_size);
+const int64_t delete_rowset_batch_size =
+std::min(50, config::recycle_rowsets_delete_batch_size);
+auto try_reserve_tablet_recycle_slot = [&](int64_t tablet_id) -> bool {
Review Comment:
Please clamp this mutable config to at least 1, similar to
`rowset_batch_size_per_tablet`. If an operator sets
`recycle_rowsets_delete_batch_size` to 0 or a negative value at runtime,
`rowsets.size() < delete_rowset_batch_size` either flushes on every iterator
for 0 or, for negative values converted in the `size_t` comparison, stays true
until the full scan finishes. In the latter case the recycler can retain every
scanned `RowsetMetaCloudPB` in `rowsets`, which defeats the memory cap and can
OOM on a large recycle backlog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4561553972 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4561552807 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4561540141 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4561539725 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
github-actions[bot] commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4560806911 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
github-actions[bot] commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4560807061 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
hello-stephen commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4486165849 # Cloud UT Coverage Report Increment line coverage `84.46% (299/354)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/5f26a57d13c194e85aed06fcc661b6cdd035224c_5f26a57d13c194e85aed06fcc661b6cdd035224c_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/5f26a57d13c194e85aed06fcc661b6cdd035224c_5f26a57d13c194e85aed06fcc661b6cdd035224c_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 78.14% (1866/2388) | | Line Coverage | 64.63% (33612/52007) | | Region Coverage | 65.26% (16640/25497) | | Branch Coverage | 55.80% (8904/15958) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
wyxxxcat commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4485716430 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [opt](recycler) Optimize rowset recycling for recycler [doris]
hello-stephen commented on PR #63295: URL: https://github.com/apache/doris/pull/63295#issuecomment-4458502541 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
