github-actions[bot] commented on code in PR #65854:
URL: https://github.com/apache/doris/pull/65854#discussion_r3621657728
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -246,10 +268,13 @@ Status CloudCumulativeCompaction::modify_rowsets() {
// calculate new cumulative point
int64_t input_cumulative_point = cloud_tablet()->cumulative_layer_point();
auto compaction_policy =
cloud_tablet()->tablet_meta()->compaction_policy();
- int64_t new_cumulative_point =
- _engine.cumu_compaction_policy(compaction_policy)
- ->new_cumulative_point(cloud_tablet(), _output_rowset,
_last_delete_version,
- input_cumulative_point);
+ int64_t new_cumulative_point = input_cumulative_point;
+ if (!_is_single_rowset_grouped_compaction) {
Review Comment:
[P1] Keep time-series level and cumulative-point transitions consistent
For a time-series tablet with level staging enabled, the first grouped
L0-to-L1 rewrite may correctly retain the cumulative point. But if its output
is still `OVERLAPPING` and above the threshold, the shortcut can select that L1
singleton again; the shared execution path promotes it to L2, while this branch
still forces the cumulative point to stay unchanged. Normal time-series
handling advances the point for an L2 output, so the persisted L2 rowset
remains in the cumulative range and later passes can inflate it further. Please
handle grouped L1+ inputs with a coherent level/point transition (or exclude
them from this shortcut), and test the repeated L1-to-L2 case.
##########
be/src/storage/rowset/vertical_beta_rowset_writer.cpp:
##########
@@ -220,6 +221,8 @@ Status VerticalBetaRowsetWriter<T>::final_flush() {
segment_writer.reset();
_record_segment_index_file_cache_preload(segment_id,
index_file_cache_info);
}
+ _segment_writers.clear();
+ _cur_writer_idx = 0;
Review Comment:
[P1] Preserve global segment-row metadata across grouped flushes
After this reset, the next segment group starts with `_cur_writer_idx == 0`,
while `_num_segment` and the output segment IDs continue globally.
`_flush_columns()` still does `_segment_num_rows.resize(_cur_writer_idx + 1)`
and writes by that group-local index, so every later group truncates/overwrites
the row counts from earlier groups. Cloud build can then persist fewer
row-count entries than `num_segments`; `RowIdConversion::add()` eventually maps
rows past the wrong destination segment (for example, on the third one-segment
group), and inverted-index compaction can reject the shortened destination
count. Please index this persistent vector by the writer's global segment ID
(without shrinking it), and cover at least three groups plus a multi-output
group.
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -520,6 +545,16 @@ Status
CloudCumulativeCompaction::pick_rowsets_to_compact() {
return st;
}
+ if (config::enable_cloud_single_rowset_compaction) {
+ for (const auto& rowset : candidate_rowsets) {
+ if (cloud::should_use_single_rowset_grouped_compaction(
Review Comment:
[P1] Prevent no-progress grouped outputs from being selected forever
The output of this branch remains immediately eligible: grouped commit
leaves the cumulative point unchanged, and more than one nonempty group marks
the result `OVERLAPPING` again. If the merge cannot reduce the segment count
below the threshold (for example, DUP segments that are already near the target
output size), the next scheduler pass selects the same version here before
consulting the normal policy, repeatedly rewrites it, and starves later
cumulative work. Please record/recognize an already-grouped no-progress result
or otherwise require demonstrable progress, and add a test that runs at least
two automatic selection cycles.
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -520,6 +545,16 @@ Status
CloudCumulativeCompaction::pick_rowsets_to_compact() {
return st;
}
+ if (config::enable_cloud_single_rowset_compaction) {
Review Comment:
[P1] Honor the active policy's NOTREADY restrictions before this shortcut
`prepare_compact()` can let a NOTREADY schema-change tablet reach this
point, but this return runs before the selected cumulative policy's lifecycle
gates. The time-series policy explicitly rejects every NOTREADY tablet, and the
size policy limits schema-change compaction to the latest ten versions; this
loop can instead choose any qualifying older rowset and replace it at the same
version. Please apply the grouped shortcut only after preserving the active
policy's tablet-state/version-window restrictions, with picker tests for both
policies.
##########
be/src/cloud/cloud_cumulative_compaction.cpp:
##########
@@ -554,6 +589,85 @@ Status
CloudCumulativeCompaction::pick_rowsets_to_compact() {
return Status::OK();
}
+Status
CloudCumulativeCompaction::prepare_merge_input_rowsets(MergeInputRowsetsResult*
result) {
+ if (!cloud::should_use_single_rowset_grouped_compaction(_input_rowsets,
*_cur_tablet_schema)) {
+ return Status::OK();
+ }
+
+ const int64_t segment_group_size =
config::cloud_single_rowset_compaction_segment_group_size;
+ if (segment_group_size <= 0) {
+ return Status::InvalidArgument(
+ "cloud_single_rowset_compaction_segment_group_size must be
positive, value={}",
+ segment_group_size);
+ }
+ _is_single_rowset_grouped_compaction = true;
+ result->is_segment_grouped = true;
+ result->segment_group_size = segment_group_size;
+ return Status::OK();
+}
+
+Status CloudCumulativeCompaction::do_merge_input_rowsets(
+ const std::vector<RowsetReaderSharedPtr>& input_rs_readers,
+ MergeInputRowsetsResult* result) {
+ if (!result->is_segment_grouped) {
+ return Compaction::do_merge_input_rowsets(input_rs_readers, result);
+ }
+
+ const int64_t segment_group_size = result->segment_group_size;
+ const auto& input_rowset = _input_rowsets.front();
+ for (int64_t segment_start = 0; segment_start <
input_rowset->num_segments();
+ segment_start += segment_group_size) {
+ const int64_t segment_end =
+ std::min(segment_start + segment_group_size,
input_rowset->num_segments());
+ const int32_t output_segment_start =
_output_rs_writer->get_allocated_segment_id();
+
+ RowsetReaderSharedPtr rs_reader;
+ RETURN_IF_ERROR(input_rowset->create_reader(&rs_reader));
+ std::vector<RowsetReaderSharedPtr> group_readers;
+ group_readers.push_back(std::move(rs_reader));
+
+ Merger::Statistics group_stats;
+ group_stats.rowid_conversion = _stats.rowid_conversion;
+ RETURN_IF_ERROR(execute_merge(group_readers, segment_end -
segment_start, &group_stats,
Review Comment:
[P2] Keep grouped-compaction progress monotonic across ranges
Every iteration here installs the same task-tracker callback, while
`vertical_merge_rowsets()` reports a complete `M/0` through `M/M` cycle for
just that range. Because `update_progress()` overwrites the counters, a task
with multiple ranges appears 100% complete after the first range and then jumps
back to 0% on the next; active-task consumers such as
`information_schema.compaction_tasks` expose these values. Please report an
aggregate total with the completed-range offset (or equivalent), and add a
two-range tracker test that reaches 100% only after the final range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]