Re: Suggested S3 FileIO/Getting Started

2020-11-12 Thread Saisai Shao
Hi all, Sorry to chime in, I also have a same concern about using Iceberg with Object storage. One of my concerns with S3FileIO is getting tied too much to a single > cloud provider. I'm wondering if an ObjectStoreFileIO would be helpful > so that S3FileIO and (a future) GCSFileIO could share

Re: Suggested S3 FileIO/Getting Started

2020-11-12 Thread Daniel Weeks
Hey John, about the concerns around cloud provider dependency, I feel like the FileIO interface is actually the right level of abstraction already. That interface basically requires "open for read" and "open for write", where the implementation will diverge across different platforms. I guess