prclin commented on PR #9524: URL: https://github.com/apache/seatunnel/pull/9524#issuecomment-3051827594
> The core problem of the current WAL logic is the lack of compaction logic. We need to introduce `.data`(Sorted String Table) files to save historical data and remove WAL files by writing WAL to data files at regular intervals.当前 WAL logic 的核心问题是缺乏 compaction logic。我们需要引入 `.data`（Sorted String Table） 文件来保存历史数据，并通过定期将 WAL 写入数据文件来删除 WAL 文件。 Your current version only filters and compresses data in WAL, which can only solve the problem of the same data appearing multiple times, but all the data needs to be traversed each time.您当前版本仅对 WAL 中的数据进行过滤和压缩，只能解决同一数据多次出现，但每次都需要遍历所有数据的问题。 Given the update frequency of metrics imap, compression is necessary, otherwise disk capacity will increase sharply; The existing method is to save a file without a key value, which avoids filtering and overall writing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
