Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-15 Thread via GitHub


airborne12 merged PR #44994:
URL: https://github.com/apache/doris/pull/44994


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-15 Thread via GitHub


github-actions[bot] commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2544422465

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-11 Thread via GitHub


doris-robot commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2535980229

   TeamCity be ut coverage result:
Function Coverage: 38.81% (10119/26071) 
Line Coverage: 29.77% (85005/285535)
Region Coverage: 28.79% (43575/151329)
Branch Coverage: 25.35% (22135/87332)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/80270e677fbe0e03547b258c0e4fbb9c36cba70f_80270e677fbe0e03547b258c0e4fbb9c36cba70f/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-11 Thread via GitHub


csun5285 commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1880109837


##
be/test/olap/rowset/segment_v2/inverted_index/common/inverted_index_gc_binlogs_test.cpp:
##
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include 
+
+#include "olap/rowset/beta_rowset_writer.h"
+#include "olap/rowset/rowset_factory.h"
+#include "olap/rowset/rowset_meta_manager.h"
+#include "olap/storage_engine.h"
+
+namespace doris {
+
+using namespace doris::vectorized;
+
+constexpr static uint32_t MAX_PATH_LEN = 1024;
+constexpr static std::string_view dest_dir = "./ut_dir/inverted_index_test";
+constexpr static std::string_view tmp_dir = "./ut_dir/tmp";
+static int64_t inc_id = 0;
+
+class IndexGcBinglogsTest : public ::testing::Test {
+protected:
+void SetUp() override {
+// absolute dir
+char buffer[MAX_PATH_LEN];
+EXPECT_NE(getcwd(buffer, MAX_PATH_LEN), nullptr);
+_curreent_dir = std::string(buffer);
+_absolute_dir = _curreent_dir + std::string(dest_dir);
+
EXPECT_TRUE(io::global_local_filesystem()->delete_directory(_absolute_dir).ok());
+
EXPECT_TRUE(io::global_local_filesystem()->create_directory(_absolute_dir).ok());
+
+// tmp dir
+
EXPECT_TRUE(io::global_local_filesystem()->delete_directory(tmp_dir).ok());
+
EXPECT_TRUE(io::global_local_filesystem()->create_directory(tmp_dir).ok());
+std::vector paths;
+paths.emplace_back(std::string(tmp_dir), 102400);
+auto tmp_file_dirs = std::make_unique(paths);
+EXPECT_TRUE(tmp_file_dirs->init().ok());
+ExecEnv::GetInstance()->set_tmp_file_dir(std::move(tmp_file_dirs));
+
+// storage engine
+doris::EngineOptions options;
+auto engine = std::make_unique(options);
+_engine_ref = engine.get();
+_data_dir = std::make_unique(*_engine_ref, _absolute_dir);
+static_cast(_data_dir->update_capacity());
+EXPECT_TRUE(_data_dir->init(true).ok());
+ExecEnv::GetInstance()->set_storage_engine(std::move(engine));
+
+// tablet_schema
+TabletSchemaPB schema_pb;
+schema_pb.set_keys_type(KeysType::DUP_KEYS);
+
schema_pb.set_inverted_index_storage_format(InvertedIndexStorageFormatPB::V2);

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-11 Thread via GitHub


csun5285 commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2535856164

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-11 Thread via GitHub


qidaye commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1879690037


##
be/test/olap/rowset/segment_v2/inverted_index/common/inverted_index_gc_binlogs_test.cpp:
##
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include 
+
+#include "olap/rowset/beta_rowset_writer.h"
+#include "olap/rowset/rowset_factory.h"
+#include "olap/rowset/rowset_meta_manager.h"
+#include "olap/storage_engine.h"
+
+namespace doris {
+
+using namespace doris::vectorized;
+
+constexpr static uint32_t MAX_PATH_LEN = 1024;
+constexpr static std::string_view dest_dir = "./ut_dir/inverted_index_test";
+constexpr static std::string_view tmp_dir = "./ut_dir/tmp";
+static int64_t inc_id = 0;
+
+class IndexGcBinglogsTest : public ::testing::Test {
+protected:
+void SetUp() override {
+// absolute dir
+char buffer[MAX_PATH_LEN];
+EXPECT_NE(getcwd(buffer, MAX_PATH_LEN), nullptr);
+_curreent_dir = std::string(buffer);
+_absolute_dir = _curreent_dir + std::string(dest_dir);
+
EXPECT_TRUE(io::global_local_filesystem()->delete_directory(_absolute_dir).ok());
+
EXPECT_TRUE(io::global_local_filesystem()->create_directory(_absolute_dir).ok());
+
+// tmp dir
+
EXPECT_TRUE(io::global_local_filesystem()->delete_directory(tmp_dir).ok());
+
EXPECT_TRUE(io::global_local_filesystem()->create_directory(tmp_dir).ok());
+std::vector paths;
+paths.emplace_back(std::string(tmp_dir), 102400);
+auto tmp_file_dirs = std::make_unique(paths);
+EXPECT_TRUE(tmp_file_dirs->init().ok());
+ExecEnv::GetInstance()->set_tmp_file_dir(std::move(tmp_file_dirs));
+
+// storage engine
+doris::EngineOptions options;
+auto engine = std::make_unique(options);
+_engine_ref = engine.get();
+_data_dir = std::make_unique(*_engine_ref, _absolute_dir);
+static_cast(_data_dir->update_capacity());
+EXPECT_TRUE(_data_dir->init(true).ok());
+ExecEnv::GetInstance()->set_storage_engine(std::move(engine));
+
+// tablet_schema
+TabletSchemaPB schema_pb;
+schema_pb.set_keys_type(KeysType::DUP_KEYS);
+
schema_pb.set_inverted_index_storage_format(InvertedIndexStorageFormatPB::V2);

Review Comment:
   Add test for v1, too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


doris-robot commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2533637521

   TeamCity be ut coverage result:
Function Coverage: 38.85% (10119/26044) 
Line Coverage: 29.77% (84946/285331)
Region Coverage: 28.81% (43560/151213)
Branch Coverage: 25.34% (22116/87270)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/a7c6538d1caf8b392d7c42ce9c1497810540bf97_a7c6538d1caf8b392d7c42ce9c1497810540bf97/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


csun5285 commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2533596478

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


github-actions[bot] commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1879312363


##
be/src/http/action/download_binlog_action.cpp:
##
@@ -144,8 +144,19 @@ void handle_get_segment_index_file(StorageEngine& engine, 
HttpRequest* req,
 const auto& rowset_id = get_http_param(req, kRowsetIdParameter);
 const auto& segment_index = get_http_param(req, 
kSegmentIndexParameter);
 const auto& segment_index_id = req->param(kSegmentIndexIdParameter);
-segment_index_file_path =
-tablet->get_segment_index_filepath(rowset_id, segment_index, 
segment_index_id);
+auto segment_file_path = tablet->get_segment_filepath(rowset_id, 
segment_index);
+if (tablet->tablet_schema()->get_inverted_index_storage_format() ==
+InvertedIndexStorageFormatPB::V1) {
+// now CCR not support for variant + index v1
+constexpr std::string_view index_suffix = "";

Review Comment:
   warning: redundant string initialization [readability-redundant-string-init]
   
   ```suggestion
   constexpr std::string_view index_suffix;
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


doris-robot commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2531471533

   TeamCity be ut coverage result:
Function Coverage: 38.86% (10121/26044) 
Line Coverage: 29.77% (84933/285320)
Region Coverage: 28.81% (43565/151206)
Branch Coverage: 25.35% (22125/87266)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/69546c74764a6b522259cae15bd4c87d76515880_69546c74764a6b522259cae15bd4c87d76515880/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


github-actions[bot] commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1877853208


##
be/test/olap/rowset/segment_v2/inverted_index/common/inverted_index_gc_binlogs_test.cpp:
##
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include 

Review Comment:
   warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]
   ```cpp
   #include 
^
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


csun5285 commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1877846497


##
be/src/olap/task/engine_storage_migration_task.cpp:
##
@@ -409,8 +409,9 @@ Status 
EngineStorageMigrationTask::_copy_index_and_data_files(
 InvertedIndexStorageFormatPB::V1) {
 for (const auto& index : tablet_schema.inverted_indexes()) {
 auto index_id = index->index_id();
-auto index_file =
-_tablet->get_segment_index_filepath(rowset_id, 
segment_index, index_id);
+auto index_file = 
InvertedIndexDescriptor::get_index_file_path_v1(
+
InvertedIndexDescriptor::get_index_file_path_prefix(segment_file_path),
+index_id, "");

Review Comment:
   no index suffix here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


csun5285 commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1877843657


##
be/src/olap/snapshot_manager.cpp:
##
@@ -700,8 +700,10 @@ Status SnapshotManager::_create_snapshot_files(const 
TabletSharedPtr& ref_tablet
 InvertedIndexStorageFormatPB::V1) {
 for (const auto& index : tablet_schema.inverted_indexes()) 
{
 auto index_id = index->index_id();
-auto index_file = 
ref_tablet->get_segment_index_filepath(
-rowset_id, segment_index, index_id);
+auto index_file = 
InvertedIndexDescriptor::get_index_file_path_v1(
+
InvertedIndexDescriptor::get_index_file_path_prefix(
+segment_file_path),
+index_id, "");

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


csun5285 commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1877840619


##
be/src/olap/task/engine_storage_migration_task.cpp:
##
@@ -409,8 +409,9 @@ Status 
EngineStorageMigrationTask::_copy_index_and_data_files(
 InvertedIndexStorageFormatPB::V1) {
 for (const auto& index : tablet_schema.inverted_indexes()) {
 auto index_id = index->index_id();
-auto index_file =
-_tablet->get_segment_index_filepath(rowset_id, 
segment_index, index_id);
+auto index_file = 
InvertedIndexDescriptor::get_index_file_path_v1(
+
InvertedIndexDescriptor::get_index_file_path_prefix(segment_file_path),
+index_id, "");

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


csun5285 commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1877840167


##
be/src/olap/tablet.cpp:
##
@@ -2653,10 +2629,22 @@ void Tablet::gc_binlogs(int64_t version) {
 
 // add binlog segment files and index files
 for (int64_t i = 0; i < num_segments; ++i) {
-
wait_for_deleted_binlog_files.emplace_back(get_segment_filepath(rowset_id, i));
-for (const auto& index : 
this->tablet_schema()->inverted_indexes()) {
-wait_for_deleted_binlog_files.emplace_back(
-get_segment_index_filepath(rowset_id, i, 
index->index_id()));
+auto segment_file_path = get_segment_filepath(rowset_id, i);
+wait_for_deleted_binlog_files.emplace_back(segment_file_path);
+
+// index files
+if (tablet_schema()->get_inverted_index_storage_format() ==
+InvertedIndexStorageFormatPB::V1) {
+for (const auto& index : tablet_schema()->inverted_indexes()) {
+auto index_file = 
InvertedIndexDescriptor::get_index_file_path_v1(
+
InvertedIndexDescriptor::get_index_file_path_prefix(segment_file_path),
+index->index_id(), "");

Review Comment:
   ok



##
be/src/olap/task/engine_storage_migration_task.cpp:
##
@@ -409,8 +409,9 @@ Status 
EngineStorageMigrationTask::_copy_index_and_data_files(
 InvertedIndexStorageFormatPB::V1) {
 for (const auto& index : tablet_schema.inverted_indexes()) {
 auto index_id = index->index_id();
-auto index_file =
-_tablet->get_segment_index_filepath(rowset_id, 
segment_index, index_id);
+auto index_file = 
InvertedIndexDescriptor::get_index_file_path_v1(
+
InvertedIndexDescriptor::get_index_file_path_prefix(segment_file_path),
+index_id, "");

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-10 Thread via GitHub


csun5285 commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2531181173

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-09 Thread via GitHub


airborne12 commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1877486298


##
be/src/olap/task/engine_storage_migration_task.cpp:
##
@@ -409,8 +409,9 @@ Status 
EngineStorageMigrationTask::_copy_index_and_data_files(
 InvertedIndexStorageFormatPB::V1) {
 for (const auto& index : tablet_schema.inverted_indexes()) {
 auto index_id = index->index_id();
-auto index_file =
-_tablet->get_segment_index_filepath(rowset_id, 
segment_index, index_id);
+auto index_file = 
InvertedIndexDescriptor::get_index_file_path_v1(
+
InvertedIndexDescriptor::get_index_file_path_prefix(segment_file_path),
+index_id, "");

Review Comment:
   use index suffix instead of "", although we disable variant index for v1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-09 Thread via GitHub


airborne12 commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1877485134


##
be/src/olap/tablet.cpp:
##
@@ -2653,10 +2629,22 @@ void Tablet::gc_binlogs(int64_t version) {
 
 // add binlog segment files and index files
 for (int64_t i = 0; i < num_segments; ++i) {
-
wait_for_deleted_binlog_files.emplace_back(get_segment_filepath(rowset_id, i));
-for (const auto& index : 
this->tablet_schema()->inverted_indexes()) {
-wait_for_deleted_binlog_files.emplace_back(
-get_segment_index_filepath(rowset_id, i, 
index->index_id()));
+auto segment_file_path = get_segment_filepath(rowset_id, i);
+wait_for_deleted_binlog_files.emplace_back(segment_file_path);
+
+// index files
+if (tablet_schema()->get_inverted_index_storage_format() ==
+InvertedIndexStorageFormatPB::V1) {
+for (const auto& index : tablet_schema()->inverted_indexes()) {
+auto index_file = 
InvertedIndexDescriptor::get_index_file_path_v1(
+
InvertedIndexDescriptor::get_index_file_path_prefix(segment_file_path),
+index->index_id(), "");

Review Comment:
   use index suffix instead of ""?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-09 Thread via GitHub


airborne12 commented on code in PR #44994:
URL: https://github.com/apache/doris/pull/44994#discussion_r1877484503


##
be/src/olap/snapshot_manager.cpp:
##
@@ -700,8 +700,10 @@ Status SnapshotManager::_create_snapshot_files(const 
TabletSharedPtr& ref_tablet
 InvertedIndexStorageFormatPB::V1) {
 for (const auto& index : tablet_schema.inverted_indexes()) 
{
 auto index_id = index->index_id();
-auto index_file = 
ref_tablet->get_segment_index_filepath(
-rowset_id, segment_index, index_id);
+auto index_file = 
InvertedIndexDescriptor::get_index_file_path_v1(
+
InvertedIndexDescriptor::get_index_file_path_prefix(
+segment_file_path),
+index_id, "");

Review Comment:
   use index suffix instead of ""?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-06 Thread via GitHub


doris-robot commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2522663210

   TeamCity be ut coverage result:
Function Coverage: 38.54% (10022/26002) 
Line Coverage: 29.57% (84102/284380)
Region Coverage: 28.65% (43188/150739)
Branch Coverage: 25.24% (21950/86978)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/7b3245ef19985d2434e544806aac30e9b976eff5_7b3245ef19985d2434e544806aac30e9b976eff5/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-06 Thread via GitHub


csun5285 commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2522539594

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-06 Thread via GitHub


github-actions[bot] commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2522473319

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-06 Thread via GitHub


github-actions[bot] commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2522473251

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-05 Thread via GitHub


doris-robot commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2522281861

   TeamCity be ut coverage result:
Function Coverage: 38.53% (10021/26005) 
Line Coverage: 29.58% (84119/284399)
Region Coverage: 28.64% (43179/150746)
Branch Coverage: 25.23% (21945/86980)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/c3ff1a52ea3fe4887af7dc32f57f9872e0bd939d_c3ff1a52ea3fe4887af7dc32f57f9872e0bd939d/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-05 Thread via GitHub


csun5285 commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2522238794

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-05 Thread via GitHub


csun5285 commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2522234141

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-04 Thread via GitHub


doris-robot commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2516659929

   TeamCity be ut coverage result:
Function Coverage: 38.48% (10007/26007) 
Line Coverage: 29.53% (83904/284176)
Region Coverage: 28.62% (43109/150610)
Branch Coverage: 25.22% (21915/86884)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/7b0c52fe747cb9749213bb354b91a3b387041af6_7b0c52fe747cb9749213bb354b91a3b387041af6/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-04 Thread via GitHub


csun5285 commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2516569622

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-04 Thread via GitHub


doris-robot commented on PR #44994:
URL: https://github.com/apache/doris/pull/44994#issuecomment-2516568965

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting 
information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it 
now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and 
after the optimization?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



[PR] [fix] (inverted index) fix index file in gc binlogs [doris]

2024-12-04 Thread via GitHub


csun5285 opened a new pull request, #44994:
URL: https://github.com/apache/doris/pull/44994

   ### What problem does this PR solve?
   
   
   
   Problem Summary:
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test 
   - [ ] Regression test
   - [ ] Unit Test
   - [ ] Manual test (add detailed scripts or steps below)
   - [ ] No need to test or manual test. Explain why:
   - [ ] This is a refactor/code format and no logic has been changed.
   - [ ] Previous test can cover this change.
   - [ ] No code files have been changed.
   - [ ] Other reason 
   
   - Behavior changed:
   - [ ] No.
   - [ ] Yes. 
   
   - Does this need documentation?
   - [ ] No.
   - [ ] Yes. 
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]