wenzhenghu opened a new issue, #51735: URL: https://github.com/apache/doris/issues/51735
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris 3.0.4 and master ### What's Wrong? The memtable-on-sink optimization can lead to uncontrolled disk space usage on BE nodes, bypassing the flood_storage constraint. when memtable-on-sink enabled, datadir disk usage is only checked on upstream be, but the real data write operation(segment file writing) is happened on downstream be which has not been checked. ### What You Expected? we should check the downstream be datadir disk usage when memtable-on-sink enabled ### How to Reproduce? 1. This conclusion can be drawn by analyzing the code. upstream side: →VTabletWriterV2::write →DeltaWriterV2::write→DeltaWriterV2::_write_memtable →MemTableWriter::write→MemTableWriter::_flush_memtable_async →FlushToken::submit→FlushToken::_flush_memtable->FlushToken::_do_flush_memtable →BetaRowsetWriterV2::flush_memtable →SegmentFlusher::flush_single_block->SegmentFlusher::add_rows/_flush_segment_writer →VerticalSegmentWriter::write_batch/VerticalSegmentWriter::finalize（will check datadir disk usage here, but it is useless） →StreamSinkFileWriter::appendv/StreamSinkFileWriter::_finalize →LoadStreamStub::append_data ->LoadStreamStub::_encode_and_send downstream side: →LoadStream::on_received_messages →LoadStream::_dispatch →IndexStream::append_data →TabletStream::append_data →LoadStreamWriter::append_data →LocalFileWriter::append has not check datadir disk usage in downstream side be node. 3. enable stream_load_default_memtable_on_sink_node, set a low flood storage* config. and launch a streamload, loading data exceed the flood config ### Anything Else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
