morningman opened a new issue, #17764:
URL: https://github.com/apache/doris/issues/17764
## Backgroud
At present, the file I/O interface of Doris is very confusing, and there are
the following problems:
- Class confusion: There are multiple sets of file I/O classes in the BE
code, such as file reader/writer, file handler, file_utils, etc. The use is
confusing and the logic is not unified.
- Interface confusion: The interface for I/O is different, and the behavior,
parameter, and return value are confusing. The usage pattern is also different.
- Hierarchical confusion:I/O classes are scattered in different directories,
which is difficult to maintain and manage.
In view of the above problems, we need to fully sort out Doris's I/O logic.
## Status quo
Currently, the following file I/O interfaces exist in Doris:
1. io/
The file system, file reader, and file writer are defined in this
directory. It is currently the most used set of io interfaces, but the levels
and interfaces are quite confusing.
2. env/
This directory defines a series of interfaces such as env/posix_env,
writable file, etc. It is almost duplicated in the io/ directory, and is also
used everywhere in the code.
3. olap/file_helper.h
It mainly defines FileHeader, but there is also a FileHandler file read
and write class.
4. util/file_utils.h filesystem_utils.h
Duplicate with filesystem under io/.
5. src/util/storage_backend
Duplicate with filesystem under io/.
## Goal
All I/O classes are unified under the io/ directory, and I/O classes in
other places are deleted.
And ensure that all file read and write logic uses the methods in the io/
directory uniformly, instead of calling system methods independently.
## New Interface
The new interface are as follows:
### 1. FileSystem
`FileSystem` is an abstract class of the file system, which defines methods
related to file operations:
```
Status create_file(const Path& file, FileWriterPtr* writer);
Status open_file(const Path& file, const FileReaderOptions& reader_options,
FileReaderSPtr* reader);
Status create_directory(const Path& dir);
Status delete_file(const Path& file);
Status delete_directory(const Path& dir);
Status batch_delete(const std::vector<Path>& files);
Status exists(const Path& path, bool* res) const;
Status file_size(const Path& file, size_t* file_size) const;
Status list(const Path& dir, bool regular_file, std::vector<FileInfo>*
files);
Status rename(const Path& orig_name, const Path& new_name);
Status rename_dir(const Path& orig_name, const Path& new_name);
```
The deriviation of each subclass below:
```
FileSystem
|---LocalFileSystem
|---RemoteFileSystem
|---S3FileSysmte
|---HdfsFileSystem
|---BrokerFileSysmte
```
`LocalFileSystem` has additional interfaces:
```
Status link_file(const Path& src, const Path& dest);
```
`RemoteFileSystem` has additional interfaces:
```
Status upload(const Path& local_file, const Path& dest_file);
Status batch_upload(const std::vector<Path>& local_files, const
std::vector<Path>& remote_files);
Status direct_upload(const Path& remote_file, const std::string& content);
Status upload_with_checksum(const Path& local_file, const Path& remote,
const std::string& checksum);
Status download(const Path& remote_file, const Path& local);
Status direct_download(const Path& remote_file, std::string* content);
Status connect();
```
The bthread and pthread issues are handled in the `FileSystem` class. The
actual IO operations are unified by `AsyncIO`. The following code example:
```
Status FileSystem::delete_file(const Path& file) {
auto path = absolute_path(file);
if (bthread_self() == 0) {
return delete_file_impl(path);
}
Status s;
auto task = [&] { s = delete_file_impl(path); };
AsyncIO::run_task(task, _type);
return s;
}
```
Subclasses only need to implement the `xxx_impl` method.
### 2. FileReader
The file reading class defines related methods for reading files.
`FileReader` must be created via `FileSystem`'s `open_file` method.
```
Status read_at(size_t offset, Slice result, size_t* bytes_read, const
IOContext* io_ctx = nullptr);
virtual Status close() = 0;
virtual const Path& path() const = 0;
virtual size_t size() const = 0;
virtual bool closed() const = 0;
virtual std::shared_ptr<FileSystem> fs() const = 0;
```
The deriviation of each subclass below:
```
FileReader
|---LocalFileReader
|---S3FileReader
|---HdfsFileReader
|---BrokerFileReader
|---StreamLoadPipe
|---FileCache
|---CachedRemoteFileReader
```
Among them, `StreamLoadPipe` is used for stream load. `FileCache` is
currently used for hot and cold separation, and `CachedRemoteFileReader` is
used for external reading. Subsequent needs to merge `FileCache` and
`CachedRemoteFileReader`.
`StreamLoadPipe` is rather special, and currently it is directly new rather
than through `FileSystem.open_file()`.
### 3. FileWriter
The file writing class defines the method of writing files:
`FileWriter` must be created via `FileSystem`'s `create_file` method.
```
virtual Status close() = 0;
virtual Status abort() = 0;
Status append(const Slice& data) { return appendv(&data, 1); }
virtual Status appendv(const Slice* data, size_t data_cnt) = 0;
virtual Status write_at(size_t offset, const Slice& data) = 0;
virtual Status finalize() = 0;
const Path& path() const { return _path; }
size_t bytes_appended() const { return _bytes_appended; }
FileSystemSPtr fs() const { return _fs; }
```
The interface definition includes both random and sequential writes.
Subclasses choose whether to implement the corresponding interface according to
different implementations.
The deriviation of each subclass below:
```
FileWriter
|---LocalFileWriter
|---S3FileWriter
|---HdfsFileWriter
|---BrokerFileWriter
```
### FileReaderOptions & IOContext
These two classes are used when creating and using a `FileReader`. The use
of these two classes is rather confusing. This refactoring is sorted out as
follows:
- FileReaderOptions
`FileReaderOptions` are used to define the behavior of `FileReader`.
Currently only includes "whether to use Cache and Cache strategy". When
**creating** `FileReader`, you need to pass in `FileReaderOptions` to determine
the specific behavior of this `FileReader`.
- IOContext
Passed in when calling the `FileReader`'s `read` method. `IOContext`
records the context information of calling this FileReader. Such as `query id`,
`statistics`, etc. Because a `FileReader` may be called by different logics and
different threads (for example, the `FileReader` of a `Segment` is shared by
multiple tasks), so it is not possible to pass in the `IOContext` when the
`FileReader` is created, but it needs to be passed in separately every time
`read()` is called.
The caller needs to ensure the life cycle of the `IOContext` during the read.
## Schedule
- [ ] Step 1: Organize the files in the `io/fs` directory, and unify the
`FileSystem`, `FileReader`, and `FileWriter` interfaces. (#17586)
- [ ] Step 2: Delete the `StorageBackend `class. (#17586)
- [ ] Step 3: Delete the `FileHandler` class. (#17586)
- [ ] Step 4: Remove the `FileUtil/FileSystemUtil` class.
- [ ] Step 5: Delete the `Env` class.
## Test
Because it is a basic change, all operations related to files need to be
tested. include:
1. Load
- Load from S3/HDFS/Broker
- Stream Load
2. Export
- Select Outfile to S3/HDFS/Broker。
- Export to S3/HDFS/Broker。
3. Query
- Query inner table
- Query external table on S3/HDFS/Broker, with Parquet/ORC/Text format
- Query external table with file cache.
4. Hot and Cold Separation
- Upload/Download of cold data
- Query on hot/cold data
5. Backup & Restore
- Backup and Restore using S3/HDFS/Broker
6. Clone
- Clone and repair between nodes
- Clone between multiple disks in a node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
