Re: Suggested S3 FileIO/Getting Started
Hey John, about the concerns around cloud provider dependency, I feel like the FileIO interface is actually the right level of abstraction already. That interface basically requires "open for read" and "open for write", where the implementation will diverge across different platforms. I guess you could think of it as S3FileIO is to FileIO what S3AFileSystem is to FileSystem (in Hadoop). You can have many different implementations that coexist. In fact, recent changes to the Catalog allow for very flexible management of FIleIO and you could even have files within a table split across multiple cloud vendors. As to the consistency questions, the list operation can be inconsistent (e.g. if a new file is created and the implementation relies on list then read, it may not see newly created objects. Iceberg does not list, so that should not be an issue). The stated read-after-write consistency is limited and does not include: - Read after overwrite - Read after delete - Read after negative cache (e.g. a GET or HEAD that occurred before the object was created). Some of those inconsistencies have caused problems in certain cases when it comes to committing data (the negative cache being the main culprit). -Dan On Wed, Nov 11, 2020 at 6:49 PM John Clara wrote: > Update: I think I'm wrong about the listing part. I think it will only > do the HEAD request. Also it seems like the consistency issue is > probably not something my team would encounter with our current jobs. > > On 2020/11/12 02:17:10, John Clara wrote: > > (Not sure if this is actually replying or just starting a new thread)> > > > > Hi Daniel,> > > > > Thanks for the response! It's very helpful and answers a lot my > questions.> > > > > A couple follow ups:> > > > > One of my concerns with S3FileIO is getting tied too much to a single > > > cloud provider. I'm wondering if an ObjectStoreFileIO would be helpful > > > so that S3FileIO and (a future) GCSFileIO could share logic? I haven't > > > looked deep enough into the S3FileIO to know how much logic is not s3 > > > specific. Maybe the FileIO interface is enough.> > > > > About consistency (no need to respond here):> > > I'm seeing that during "getFileStatus" my version of s3a does some > list > > > requests (but I'm not sure if that could fail from consistency issues).> > > I'm also confused about the read-after-(initial) write part:> > > "Amazon S3 provides read-after-write consistency for PUTS of new > objects > > > in your S3 bucket in all Regions with one caveat. The caveat is that > if > > > you make a HEAD or GET request to a key name before the object is > > > created, then create the object shortly after that, a subsequent GET > > > might not return the object due to eventual consistency. - > > > https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html";> > > > > When my version of s3a does a create, it first does a > getMetadataRequest > > > (HEAD) to check if the object exists before creating the object. I > think > > > this is talked about in this issue: > > > https://github.com/apache/iceberg/issues/1398 and talked about in the > > > S3FileIO PR: https://github.com/apache/iceberg/pull/1573. I'll follow > up > > > in that issue for more info.> > > > > John> > > > > > > On 2020/11/12 00:36:10, Daniel Weeks > wrote:> > > > Hey John, I might be able to help answer some of your questions and > > > provide>> > > > some context around how you might want to go forward.>> > > >> > > > So, one fundamental aspect of Iceberg is that it only relies on a > few>> > > > operations (as defined by the FileIO interface). This makes much of > the>> > > > functionality and complexity of full file system implementations>> > > > unnecessary. You should not need features like S3Guard or > additional S3>> > > > operations these implementations rely on in order to achieve file > > > system>> > > > contract behavior. Consistency issues should also not be a problem > > > since>> > > > Iceberg does not overwrite or list and read-after-(initial)write is > a>> > > > guarantee provided by S3.>> > > >> > > > At Netflix, we use a custom FileSystem implementation (somewhat like > > > S3A),>> > > > but with much of the contract behavior that drives additional > > > operations>> > > > against S3 disabled. However, we are transitioning to a more native>> > > > implementation of S3FileIO, which you'll see as part of the ongoing > > > work in>> > > > Iceberg.>> > > >> > > > Per your specific questions:>> > > >> > > > 1) The S3FileIO implementation is very new, though internally we > have>> > > > something very similar. There are features missing that we are > > > working to>> > > > add (e.g. progressive multipart upload for large files is likely the > > > most>> > > > important).>> > > > 2) You can use S3AFileSystem with the HadoopFileIO implementation, > > > though>> > > > you may still see similar behavior with additional calls bein
Re: Suggested S3 FileIO/Getting Started
Hi all, Sorry to chime in, I also have a same concern about using Iceberg with Object storage. One of my concerns with S3FileIO is getting tied too much to a single > cloud provider. I'm wondering if an ObjectStoreFileIO would be helpful > so that S3FileIO and (a future) GCSFileIO could share logic? I haven't > looked deep enough into the S3FileIO to know how much logic is not s3 > specific. Maybe the FileIO interface is enough. > Now we have a S3 specific FileIO implementation, is it recommended to use this one instead of s3a like HCFS implementation? Also each public cloud provider has its own HCFS implementation for its own object storage. Are we going to suggest to create a specific FileIO implementation or use the existing HCFS implementation? Thanks Saisai Daniel Weeks 于2020年11月13日周五 上午1:09写道： > Hey John, about the concerns around cloud provider dependency, I feel like > the FileIO interface is actually the right level of abstraction already. > > That interface basically requires "open for read" and "open for write", > where the implementation will diverge across different platforms. > > I guess you could think of it as S3FileIO is to FileIO what S3AFileSystem > is to FileSystem (in Hadoop). You can have many different implementations > that coexist. > > In fact, recent changes to the Catalog allow for very flexible management > of FIleIO and you could even have files within a table split across > multiple cloud vendors. > > As to the consistency questions, the list operation can be inconsistent > (e.g. if a new file is created and the implementation relies on list then > read, it may not see newly created objects. Iceberg does not list, so that > should not be an issue). > > The stated read-after-write consistency is limited and does not include: > - Read after overwrite > - Read after delete > - Read after negative cache (e.g. a GET or HEAD that occurred before the > object was created). > > Some of those inconsistencies have caused problems in certain cases when > it comes to committing data (the negative cache being the main culprit). > > -Dan > > > On Wed, Nov 11, 2020 at 6:49 PM John Clara > wrote: > >> Update: I think I'm wrong about the listing part. I think it will only >> do the HEAD request. Also it seems like the consistency issue is >> probably not something my team would encounter with our current jobs. >> >> On 2020/11/12 02:17:10, John Clara wrote: >> > (Not sure if this is actually replying or just starting a new thread)> >> > >> > Hi Daniel,> >> > >> > Thanks for the response! It's very helpful and answers a lot my >> questions.> >> > >> > A couple follow ups:> >> > >> > One of my concerns with S3FileIO is getting tied too much to a single > >> > cloud provider. I'm wondering if an ObjectStoreFileIO would be helpful >> > >> > so that S3FileIO and (a future) GCSFileIO could share logic? I haven't >> > >> > looked deep enough into the S3FileIO to know how much logic is not s3 > >> > specific. Maybe the FileIO interface is enough.> >> > >> > About consistency (no need to respond here):> >> > I'm seeing that during "getFileStatus" my version of s3a does some >> list > >> > requests (but I'm not sure if that could fail from consistency >> issues).> >> > I'm also confused about the read-after-(initial) write part:> >> > "Amazon S3 provides read-after-write consistency for PUTS of new >> objects > >> > in your S3 bucket in all Regions with one caveat. The caveat is that >> if > >> > you make a HEAD or GET request to a key name before the object is > >> > created, then create the object shortly after that, a subsequent GET > >> > might not return the object due to eventual consistency. - > >> > https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html";> >> > >> > When my version of s3a does a create, it first does a >> getMetadataRequest > >> > (HEAD) to check if the object exists before creating the object. I >> think > >> > this is talked about in this issue: > >> > https://github.com/apache/iceberg/issues/1398 and talked about in the >> > >> > S3FileIO PR: https://github.com/apache/iceberg/pull/1573. I'll follow >> up > >> > in that issue for more info.> >> > >> > John> >> > >> > >> > On 2020/11/12 00:36:10, Daniel Weeks >> wrote:> >> > > Hey John, I might be able to help answer some of your questions and > >> > provide>> >> > > some context around how you might want to go forward.>> >> > >> >> > > So, one fundamental aspect of Iceberg is that it only relies on a >> few>> >> > > operations (as defined by the FileIO interface). This makes much of >> the>> >> > > functionality and complexity of full file system implementations>> >> > > unnecessary. You should not need features like S3Guard or >> additional S3>> >> > > operations these implementations rely on in order to achieve file > >> > system>> >> > > contract behavior. Consistency issues should also not be a problem > >> > since>> >> > > Iceberg does not overwri
