Thanks a lot Ryan for your explanation, greatly helpful. Best regards, Saisai
Ryan Blue <rb...@netflix.com.invalid> 于2020年11月14日周六 上午2:03写道： > Saisai, > > Iceberg's FileIO interface doesn't require guarantees as strict as a > Hadoop-compatible FileSystem. For S3, that allows us to avoid negative > caching that can cause problems when reading a table that has just been > updated. (Specifically, S3A performs a HEAD request to check whether a path > is a directory even for overwrites.) > > It's up to users whether they want to use S3FileIO or S3A with > HadoopFileIO, or even HadoopFileIO with a custom FileSystem. We are working > to make it easy to switch between them, but there are notable problems like > ORC not using the FileIO interface yet. I would recommend using S3FileIO to > avoid negative caching, but S3A with S3Guard may be another way to avoid > the problem. I think the default should be to use S3FileIO if it is in the > classpath, and fall back to a Hadoop FileSystem if it isn't. > > For cloud providers, I think the question is whether there is a need for > the relaxed guarantees. If there is no need, then using HadoopFileIO and a > FileSystem works great and is probably the easiest way to maintain an > implementation for the object store. > > On Thu, Nov 12, 2020 at 7:31 PM Saisai Shao <sai.sai.s...@gmail.com> > wrote: > >> Hi all, >> >> Sorry to chime in, I also have a same concern about using Iceberg with >> Object storage. >> >> One of my concerns with S3FileIO is getting tied too much to a single >>> cloud provider. I'm wondering if an ObjectStoreFileIO would be helpful >>> so that S3FileIO and (a future) GCSFileIO could share logic? I haven't >>> looked deep enough into the S3FileIO to know how much logic is not s3 >>> specific. Maybe the FileIO interface is enough. >>> >> >> Now we have a S3 specific FileIO implementation, is it recommended to use >> this one instead of s3a like HCFS implementation? Also each public cloud >> provider has its own HCFS implementation for its own object storage. Are we >> going to suggest to create a specific FileIO implementation or use the >> existing HCFS implementation? >> >> Thanks >> Saisai >> >> >> Daniel Weeks <dwe...@netflix.com.invalid> 于2020年11月13日周五 上午1:09写道： >> >>> Hey John, about the concerns around cloud provider dependency, I feel >>> like the FileIO interface is actually the right level of abstraction >>> already. >>> >>> That interface basically requires "open for read" and "open for write", >>> where the implementation will diverge across different platforms. >>> >>> I guess you could think of it as S3FileIO is to FileIO what >>> S3AFileSystem is to FileSystem (in Hadoop). You can have many different >>> implementations that coexist. >>> >>> In fact, recent changes to the Catalog allow for very flexible >>> management of FIleIO and you could even have files within a table split >>> across multiple cloud vendors. >>> >>> As to the consistency questions, the list operation can be inconsistent >>> (e.g. if a new file is created and the implementation relies on list then >>> read, it may not see newly created objects. Iceberg does not list, so that >>> should not be an issue). >>> >>> The stated read-after-write consistency is limited and does not include: >>> - Read after overwrite >>> - Read after delete >>> - Read after negative cache (e.g. a GET or HEAD that occurred before >>> the object was created). >>> >>> Some of those inconsistencies have caused problems in certain cases when >>> it comes to committing data (the negative cache being the main culprit). >>> >>> -Dan >>> >>> >>> On Wed, Nov 11, 2020 at 6:49 PM John Clara <john.anthony.cl...@gmail.com> >>> wrote: >>> >>>> Update: I think I'm wrong about the listing part. I think it will only >>>> do the HEAD request. Also it seems like the consistency issue is >>>> probably not something my team would encounter with our current jobs. >>>> >>>> On 2020/11/12 02:17:10, John Clara <j...@gmail.com> wrote: >>>> > (Not sure if this is actually replying or just starting a new >>>> thread)> >>>> > >>>> > Hi Daniel,> >>>> > >>>> > Thanks for the response! It's very helpful and answers a lot my >>>> questions.> >>>> > >>>> > A couple follow ups:> >>>> > >>>> > One of my concerns with S3FileIO is getting tied too much to a >>>> single > >>>> > cloud provider. I'm wondering if an ObjectStoreFileIO would be >>>> helpful > >>>> > so that S3FileIO and (a future) GCSFileIO could share logic? I >>>> haven't > >>>> > looked deep enough into the S3FileIO to know how much logic is not >>>> s3 > >>>> > specific. Maybe the FileIO interface is enough.> >>>> > >>>> > About consistency (no need to respond here):> >>>> > I'm seeing that during "getFileStatus" my version of s3a does some >>>> list > >>>> > requests (but I'm not sure if that could fail from consistency >>>> issues).> >>>> > I'm also confused about the read-after-(initial) write part:> >>>> > "Amazon S3 provides read-after-write consistency for PUTS of new >>>> objects > >>>> > in your S3 bucket in all Regions with one caveat. The caveat is that >>>> if > >>>> > you make a HEAD or GET request to a key name before the object is > >>>> > created, then create the object shortly after that, a subsequent GET >>>> > >>>> > might not return the object due to eventual consistency. - > >>>> > https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html";> >>>> > >>>> > When my version of s3a does a create, it first does a >>>> getMetadataRequest > >>>> > (HEAD) to check if the object exists before creating the object. I >>>> think > >>>> > this is talked about in this issue: > >>>> > https://github.com/apache/iceberg/issues/1398 and talked about in >>>> the > >>>> > S3FileIO PR: https://github.com/apache/iceberg/pull/1573. I'll >>>> follow >>>> up > >>>> > in that issue for more info.> >>>> > >>>> > John> >>>> > >>>> > >>>> > On 2020/11/12 00:36:10, Daniel Weeks <d....@netflix.com.INVALID> >>>> wrote:> >>>> > > Hey John, I might be able to help answer some of your questions >>>> and > >>>> > provide>> >>>> > > some context around how you might want to go forward.>> >>>> > >> >>>> > > So, one fundamental aspect of Iceberg is that it only relies on a >>>> few>> >>>> > > operations (as defined by the FileIO interface). This makes much >>>> of >>>> the>> >>>> > > functionality and complexity of full file system implementations>> >>>> > > unnecessary. You should not need features like S3Guard or >>>> additional S3>> >>>> > > operations these implementations rely on in order to achieve file > >>>> > system>> >>>> > > contract behavior. Consistency issues should also not be a problem >>>> > >>>> > since>> >>>> > > Iceberg does not overwrite or list and read-after-(initial)write >>>> is >>>> a>> >>>> > > guarantee provided by S3.>> >>>> > >> >>>> > > At Netflix, we use a custom FileSystem implementation (somewhat >>>> like > >>>> > S3A),>> >>>> > > but with much of the contract behavior that drives additional > >>>> > operations>> >>>> > > against S3 disabled. However, we are transitioning to a more >>>> native>> >>>> > > implementation of S3FileIO, which you'll see as part of the >>>> ongoing > >>>> > work in>> >>>> > > Iceberg.>> >>>> > >> >>>> > > Per your specific questions:>> >>>> > >> >>>> > > 1) The S3FileIO implementation is very new, though internally we >>>> have>> >>>> > > something very similar. There are features missing that we are > >>>> > working to>> >>>> > > add (e.g. progressive multipart upload for large files is likely >>>> the > >>>> > most>> >>>> > > important).>> >>>> > > 2) You can use S3AFileSystem with the HadoopFileIO implementation, >>>> > >>>> > though>> >>>> > > you may still see similar behavior with additional calls being >>>> made >>>> (I>> >>>> > > don't know if these can be disabled).>> >>>> > > 3) The PrestoS3FileSystem is tailored to Presto's use and is >>>> likely > >>>> > not as>> >>>> > > complete as S3A, but seeing as it is using the Hadoop FileSystem >>>> api, > >>>> > it>> >>>> > > would likely work for what HadoopFileIO exercises (as would the>> >>>> > > EMRFileSystem).>> >>>> > > 4) I would probably discourage you from writing your own file >>>> system > >>>> > as the>> >>>> > > S3FileIO will likely be a more optimized implementation for what > >>>> > Iceberg>> >>>> > > needs.>> >>>> > >> >>>> > > If you want to contribute or have time to help contribute to > >>>> > S3FileIO, that>> >>>> > > is the path I would recommend. As for configuration, I would say a >>>> > >>>> > lot of>> >>>> > > it comes down to how to configure the AWS S3 Client that you >>>> provide > >>>> > to the>> >>>> > > S3FileIO implementation, but a lot of the defaults are reasonable >>>> (you>> >>>> > > might want to tweak a few like max connections and maybe the retry >>>> > >>>> > policy).>> >>>> > >> >>>> > > The recently committed work to dynamically load your FileIO should >>>> > >>>> > make it>> >>>> > > relatively easy to test out and we'd love to have extra eyes and > >>>> > feedback>> >>>> > > on it.>> >>>> > >> >>>> > > Let me know if that helps,>> >>>> > > -Dan>> >>>> > >> >>>> > >> >>>> > >> >>>> > > On Wed, Nov 11, 2020 at 1:45 PM John Clara <jo...@gmail.com>>> >>>> > > wrote:>> >>>> > >> >>>> > > > Hello all,>> >>>> > > >>> >>>> > > > Thank you all for creating/continuing this great project! I am >>>> just>> >>>> > > > starting to get comfortable with the fundamentals and I'm >>>> thinking > >>>> > that my>> >>>> > > > team has been using Iceberg the wrong way at the FileIO level.>> >>>> > > >>> >>>> > > > I was wondering if people would be willing to share how they set >>>> up > >>>> > their>> >>>> > > > FileIO/FileSystem with S3 and any customizations they had to >>>> add.>> >>>> > > >>> >>>> > > > (Preferably from smaller teams. My team is small and cannot>> >>>> > > > realistically customize everything. If there's an up to date >>>> thread>> >>>> > > > discussing this that I missed, please link me that instead.)>> >>>> > > >>> >>>> > > > ***** My team's specific problems/setup which you can ignore >>>> ***>> >>>> > > >>> >>>> > > > My team has been using Hadoop FileIO with the S3AFileSystem. >>>> Jars >>>> are>> >>>> > > > provided by AWS EMR 5.23 which is on Hadoop 2.8.5. We use >>>> DynamoDB > >>>> > for>> >>>> > > > atomic renames by implementing Iceberg's provided interfaces. We >>>> > >>>> > read/write>> >>>> > > > from either Spark in EMR or on-prem JVM's in docker containers > >>>> > (managed by>> >>>> > > > k8s). Both use s3a, but the EMR clusters have HDFS (backed by >>>> core > >>>> > nodes)>> >>>> > > > for the s3a buffered writes while the on-prem containers use the >>>> > >>>> > docker>> >>>> > > > container's default file system which uses an overlay2 storage > >>>> > driver (that>> >>>> > > > I know nothing about).>> >>>> > > >>> >>>> > > > Hadoop 2.8.5's S3AFileSystem does a bunch of unnecessary get and >>>> list>> >>>> > > > requests which is well known in the community (but not to my >>>> team>> >>>> > > > unfortunately). There's also GET PUT GET inconsistency issues >>>> with > >>>> > S3 that>> >>>> > > > have been talked about, but I don't yet understand how they >>>> arise > >>>> > in the>> >>>> > > > 2.8.5 S3AFilesystem >>>> (https://github.com/apache/iceberg/issues/1398).>> >>>> > > >>> >>>> > > > *** End of specific ***>> >>>> > > >>> >>>> > > >>> >>>> > > > The options I'm seeing are:>> >>>> > > >>> >>>> > > > 1. Using Iceberg's new S3 FileIO. Is anyone using this in prod?>> >>>> > > >>> >>>> > > > This still seems very new unless it is actually based on >>>> Netflix's>> >>>> > > > prod implementation that they're releasing to the community? >>>> (I'm > >>>> > wondering>> >>>> > > > if it's safe to start moving onto it in prod in the near term. >>>> If > >>>> > Netflix>> >>>> > > > is using it (or rolling it out) that would be more than enough >>>> for > >>>> > my team.)>> >>>> > > >>> >>>> > > > 2. Using a newer hadoop version and use the S3AFileSystem. Any>> >>>> > > > recommendations on a version and are you also using S3Guard?>> >>>> > > >>> >>>> > > > From a quick look, most gains compared to older versions seem to >>>> be>> >>>> > > > from S3Guard. Are there substantial gains without it? (My team > >>>> > doesn't have>> >>>> > > > experience with S3Guard and Iceberg seems to not need it outside >>>> of > >>>> > atomic>> >>>> > > > renames?)>> >>>> > > >>> >>>> > > > 3. Using an alternative hadoop file system. Any >>>> recommendations?>> >>>> > > >>> >>>> > > > In the recent Iceberg S3 FileIO, the License states it was based >>>> off>> >>>> > > > the Presto FileSystem. Has anyone used this file system as is >>>> with > >>>> > Iceberg?>> >>>> > > > (https://github.com/apache/iceberg/blob/master/LICENSE#L251)>> >>>> > > >>> >>>> > > > 4. Roll our own hadoop file system. Anyone have stories/blogs >>>> about>> >>>> > > > pitfalls or difficulties?>> >>>> > > >>> >>>> > > > rdblue hints that Netflix already done this:>> >>>> > > > > >>>> > https://github.com/apache/iceberg/issues/1398#issuecomment-682837392 >>>> .>> >>>> > > > (My team probably doesn't have the capacity for this)>> >>>> > > >>> >>>> > > >>> >>>> > > > Places where I tried looking for this info:>> >>>> > > >>> >>>> > > > - https://github.com/apache/iceberg/issues/761 (issue for >>>> getting>> >>>> > > > started guide)>> >>>> > > > - https://iceberg.apache.org/spec/#file-system-operations>> >>>> > > >>> >>>> > > > Thanks everyone,>> >>>> > > >>> >>>> > > > John Clara>> >>>> > > >>> >>>> > >> >>>> > >>>> >>> > > -- > Ryan Blue > Software Engineer > Netflix >
