kaijchen opened a new issue, #20445: URL: https://github.com/apache/doris/issues/20445
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master (f0b0bda04a8dd758ea9e9ca9c252a3126e258667) ### What's Wrong? INSERT INTO MoW tables failed, because the segment id for generate delete bitmap overlaps. Also, generate delete bitmap should be done before segment compaction, otherwise the original segment may be deleted. ``` I0605 12:56:04.091609 2489391 beta_rowset_writer.cpp:826] [debug] tablet_id: 38328, flushed segment id 15, num_segment 16 I0605 12:56:04.091657 2489391 memtable.cpp:448] [debug] tablet id: 38328 num segment before: 13, after: 16 I0605 12:56:04.410606 2489639 beta_rowset_writer.h:113] [debug] tablet id: 38328, allocated segment id: 17 I0605 12:56:04.549297 2489434 segcompaction.cpp:288] segcompaction completed. tablet_id:38328 rowset_id:0200000000001ff04d4a8f1729e29387beaa22ada823e49c elapsed time:6076413us. update segcompacted_point:4 segment num:2 begin:2 end:3 I0605 12:56:04.574409 2489385 beta_rowset_writer.cpp:788] [debug] tablet_id: 38328 flushing filename: /root/testload/doris-debug/be/storage rowset_id:0200000000001ff04d4a8f1729e29387beaa22ada823e49c segment id: 16 segment num:16 I0605 12:56:04.574488 2489385 beta_rowset_writer.cpp:826] [debug] tablet_id: 38328, flushed segment id 16, num_segment 17 I0605 12:56:04.576594 2489385 memtable.cpp:448] [debug] tablet id: 38328 num segment before: 15, after: 17 W0605 12:56:04.576639 2489385 file_system.cpp:33] [IO_ERROR]failed to get file size /root/testload/doris-debug/be/storage/data/904/38328/193175211/0200000000001ff04d4a8f1729e29387beaa22ada823e49c_15.dat: (2), No such file or directory W0605 12:56:04.593124 2489385 beta_rowset.cpp:157] failed to open segment. /root/testload/doris-debug/be/storage/data/904/38328/193175211/0200000000001ff04d4a8f1729e29387beaa22ada823e49c_15.dat under rowset /root/testload/doris-debug/be/storage/data/904/38328/193175211/0200000000001ff04d4a8f1729e29387beaa22ada823e49c : [IO_ERROR]failed to get file size /root/testload/doris-debug/be/storage/data/904/38328/193175211/0200000000001ff04d4a8f1729e29387beaa22ada823e49c_15.dat: (2), No such file or directory W0605 12:56:04.593179 2489385 memtable_flush_executor.cpp:100] Flush memtable failed with res = [IO_ERROR]failed to get file size /root/testload/doris-debug/be/storage/data/904/38328/193175211/0200000000001ff04d4a8f1729e29387beaa22ada823e49c_15.dat: (2), No such file or directory ``` ``` I0605 15:14:01.832816 1434878 beta_rowset_writer.cpp:826] [debug] tablet_id: 46976, flushed segment id 18, num_segment 19 I0605 15:14:01.832968 1434878 memtable.cpp:448] [debug] tablet id: 46976 num segment before: 17, after: 19 I0605 15:14:01.985030 1435116 beta_rowset_writer.h:113] [debug] tablet id: 46976, allocated segment id: 20 I0605 15:14:02.170238 1434878 tablet.cpp:2903] construct delete bitmap tablet: 46976 rowset: 0200000000002049a74e86ee34027b145e82d74e867d71b4 dummy_version: [2-2]bitmap num: 0 cost: 337193(us) I0605 15:14:02.314808 1434334 daemon.cpp:213] OS physical memory 61.33 GB. Process memory usage 25.10 GB, limit 49.06 GB, soft limit 44.15 GB. Sys available memory 40.12 GB, low water mark 1.60 GB, warning water mark 3.20 GB. Refresh interval memory growth 0 B I0605 15:14:02.495476 1434880 beta_rowset_writer.cpp:788] [debug] tablet_id: 46976 flushing dir: /root/testload/doris-debug/be/storage rowset_id:0200000000002049a74e86ee34027b145e82d74e867d71b4 segment id: 19 segment num:19 I0605 15:14:02.496124 1434880 beta_rowset_writer.cpp:826] [debug] tablet_id: 46976, flushed segment id 19, num_segment 20 I0605 15:14:02.497874 1434880 memtable.cpp:448] [debug] tablet id: 46976 num segment before: 18, after: 20 W0605 15:14:02.497910 1434880 file_system.cpp:33] [IO_ERROR]failed to get file size /root/testload/doris-debug/be/storage/data/955/46976/1585439871/0200000000002049a74e86ee34027b145e82d74e867d71b4_18.dat: (2), No such file or directory W0605 15:14:02.498517 1434880 beta_rowset.cpp:157] failed to open segment. /root/testload/doris-debug/be/storage/data/955/46976/1585439871/0200000000002049a74e86ee34027b145e82d74e867d71b4_18.dat under rowset /root/testload/doris-debug/be/storage/data/955/46976/1585439871/0200000000002049a74e86ee34027b145e82d74e867d71b4 : [IO_ERROR]failed to get file size /root/testload/doris-debug/be/storage/data/955/46976/1585439871/0200000000002049a74e86ee34027b145e82d74e867d71b4_18.dat: (2), No such file or directory W0605 15:14:02.498538 1434880 memtable_flush_executor.cpp:100] Flush memtable failed with res = [IO_ERROR]failed to get file size /root/testload/doris-debug/be/storage/data/955/46976/1585439871/0200000000002049a74e86ee34027b145e82d74e867d71b4_18.dat: (2), No such file or directory ``` ### What You Expected? The load should success. ### How to Reproduce? Run `INSERT INTO mowtable SELECT * FROM duptable` multiple times (with large amout of data). ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
