yangzhg opened a new issue #5232:
URL: https://github.com/apache/incubator-doris/issues/5232
Using broker to read hdfs and bos Although the implementation of different
file systems is shielded, there are the following problems:
1. The process of reading and writing interacts with the broker too much
through the thrift interface, which increases the probability of errors
2. The performance of reading via broker is worse than direct reading due
to transit
3. The method of uploading files through the broker is to first upload to
the temporary file and then rename it into the final file. In many cloud
storages, renaming is not supported. Some of the methods of copy+delete are
real copying rather than simple modification. Data is a waste of time,
especially when there are more files to upload
Therefore, we intend to support direct access through hdfs and s3 protocols,
and support most cloud storage through s3 protocols, such as baidu bos, aliyun
oss, etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
