Xuanwo commented on issue #2367: URL: https://github.com/apache/datafusion-comet/issues/2367#issuecomment-3625693588
Thank you for starting this! > Do we still need to pursue the fs-hdfs approach or can we replace it with OpenDAL I 100% support replacing with OpenDAL > How does OpenDAL support s3? Yes > Does it use a Rust-native solution using the AWS Rust SDK, or does it use the same approach as HDFS support where it uses hadoop-aws in JVM? Build it by talking directly to the HTTP API from scratch. To ensure its correctness, we test against different kinds of S3 services. <img width="810" height="574" alt="Image" src="https://github.com/user-attachments/assets/4d24cceb-68f1-44b4-807d-5da97d8d69b4"; /> > Can OpenDAL support all of our requirements around custom authentication? Yes. OpenDAL is powered by `reqsign` and allow users to implement their own authentication <img width="1319" height="964" alt="Image" src="https://github.com/user-attachments/assets/f3270c7b-c917-4901-a29e-fc040b2b6270"; /> > Does OpenDAL perform well? Yes. Already adopted by many databases. It's built on zero-cost principles so users don't have to pay extra for things they don't need. And opendal also provides the best native concurrent read/write API like: ```rust // download file with 16 concurrent requests with 4MiB as chunk. let data = op.read_with(path).chunk(4 * 1024 * 1024).concurrent(16).await?; // upload file with 4 concurrent requests with 16MiB as chunk. let _ = op.write_with(path, data).chunk(16 * 1024 * 1024).concurrent(4).await?; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
