1. I agree, this is what the spec currently requires 2. I agree, it's up for consideration
3. I agree, I think if an implementation didn't adhere to the current spec requirements, I would say it's out of spec (not sure I'd go as far as to say it's a different kind of table entirely). Just to expand on #3, we will find lots of cases where implementations deviate (likely unintentionally) from the rest/table spec and I feel like the goal is to identify those cases and steer them back into compliance with the spec as opposed to immediately claiming they're something entirely different. The overarching goal is to improve openness and interoperability. My main point is that there isn't an inherent incompatibility between the REST spec and the Iceberg spec. The preservation of the storage representation was discussed and intentional during the design/development of the REST spec. -Dan On Thu, Feb 29, 2024 at 3:40 PM Jack Ye <yezhao...@gmail.com> wrote: > > For example, I cannot validate the atomic behaviors Glue claims, but I > wouldn't assert that it is non-compliant because of that. > > I think these are not comparable claims because the API scope is > completely different, but I don't think it's worth arguing in depth. Let's > try to see if we can have some consensus. > > Based on what you said above, do you agree with the following 3 points? > > 1. Today, a table/view in any catalog including a REST spec-compatible > catalog is an Iceberg table/view if and only if it points to a JSON > metadata file in storage. This concept is a part of the Iceberg table/view > spec. There is a debate to be had for if we want to remove this requirement > or not. The argument for it (as Yufei said) is to use other storage for > better performance. The argument against it (as Amogh said) is to keep > Iceberg open source friendly through the JSON format. > > 2. Today, a table/view in any catalog including a REST spec-compatible > catalog is an Iceberg table/view if and only if it behind the scene > performs the atomic metadata file swap for every commit. This concept is a > part of the Iceberg table/view spec. We should consider removing this > requirement in the Iceberg table/view spec. > > 3. A table/view in an Iceberg REST spec-compatible catalog may or may not > be an Iceberg table/view. The REST spec does not enforce this, and this > stance will remain true going forward. For example, it could use the > Iceberg table/view metadata structure but does not store the metadata in > JSON file, or not use the metadata file swap commit procedure, or both, and > in those cases it is not an Iceberg table/view. More extremely, it might be > a totally different kind of table that is only surfaced through the REST > models. > > -Jack > > On Thu, Feb 29, 2024 at 2:13 PM Daniel Weeks <daniel.c.we...@gmail.com> > wrote: > >> > In that case are tables in a REST-compliant catalog still an Iceberg >> table? I don't think so, because it is a table that only partially follows >> the Iceberg table spec. >> >> If the catalog is REST compliant and complies with the Iceberg spec, they >> are still Iceberg tables. I can see there is an argument that if the >> catalog is REST compliant but does not follow the commit requirements (or >> aspects of the Iceberg spec), that you cannot call those Iceberg tables. >> But the assertion that Iceberg tables in a REST catalog are de facto >> non-compliant is incorrect. >> >> > I like the idea about validation for format compliance. But don't think >> you can technically validate this. You can validate the static table to see >> if it has all the Iceberg metadata components, but you can not validate the >> internal behavior of the service during a commit to see if it really >> atomically swapped a metadata file. >> >> Just because you cannot see/validate the implementation doesn't mean that >> it is non-compliant. For example, I cannot validate the atomic behaviors >> Glue claims, but I wouldn't assert that it is non-compliant because of that. >> >> I do think there is a discussion to be had about if/when we might adjust >> the storage/swap requirements, but to reinforce Amogh's point, removing >> those requirements would impact the openness and accessibility of Iceberg, >> which I feel would hamper adoption. >> >> -Dan >> >> >> >> On Thu, Feb 29, 2024 at 1:53 PM Yufei Gu <flyrain...@gmail.com> wrote: >> >>> We've periodically discussed removing the storage requirement and I >>>> think there's a path forward to do that and would agree that standardizing >>>> on REST, but I wouldn't say the justification for making this push is that >>>> REST is not compliant so we can just ignore the table spec requirements. >>>> There are a few more things to consider, which is that not everything >>>> can use REST currently and making a hard cut away from file based metadata >>>> could bifurcate access to Iceberg data. There are also aspects to the spec >>>> that reference the metadata paths (like metadata log, though it's >>>> optional), but would likely need to be addressed. >>> >>> >>> This is a bit off-topic. It makes sense to me to remove the storage >>> requirement moving foward. The metadata.json file isn't necessary in the >>> Rest catalog. For example, the rest catalog may not have the permission to >>> write to the table owner's storage. It still can save it as a file of >>> course, but doesn't quite make sense. Putting it in a key-value store or >>> RDMS could be a better option. >>> >>> Given that we are going to remove the storage requirement. Should we >>> avoid the file path in the current design for things like view spec? A >>> solution like table identifier + version uuid may serve the purpose. >>> >>> Yufei >>> >>> >>> On Thu, Feb 29, 2024 at 1:29 PM Jack Ye <yezhao...@gmail.com> wrote: >>> >>>> > There's no exemption that says if you're using REST you don't need to >>>> follow the spec. Why do you think that's the case? >>>> >>>> In that case are tables in a REST-compliant catalog still an Iceberg >>>> table? I don't think so, because it is a table that only partially follows >>>> the Iceberg table spec. >>>> >>>> I like the idea about validation for format compliance. But don't think >>>> you can technically validate this. You can validate the static table to see >>>> if it has all the Iceberg metadata components, but you can not validate the >>>> internal behavior of the service during a commit to see if it really >>>> atomically swapped a metadata file. >>>> >>>> So I think at minimum we should update the table/view spec to remove >>>> the metadata file swap requirement. The Iceberg table/view spec should be a >>>> pure format spec that specifies how the file is laid out in storage. >>>> >>>> -Jack >>>> >>>> On Thu, Feb 29, 2024 at 1:22 PM Amogh Jahagirdar <am...@tabular.io> >>>> wrote: >>>> >>>>> I want to echo Dan's point that just because there is a separate spec >>>>> for a REST Catalog does not mean that implementations can deviate from the >>>>> spec's definition of the commit protocol or metadata layout, and still be >>>>> considered "spec compliant". >>>>> >>>>> > Secondly, once we do that, we should declare REST spec as the >>>>> official catalog spec to interact with Iceberg tables. Otherwise at least >>>>> I >>>>> will be very tempted to just break the atomic pointer swap pattern and >>>>> store the entire metadata using the Glue Table object to achieve much >>>>> better performance and also Glue native feature integrations, and I think >>>>> other players will be equally motivated to do something similar. That will >>>>> lead to even more chaos in the Iceberg catalog space. >>>>> >>>>> On this, a second point I want to make is around the openness of this >>>>> ecosystem. We all already know that openness (the file formats, the >>>>> metadata layout, the spec itself) is a fundamental tenant of the project. >>>>> If we take the provided example of removing the metadata JSON file and >>>>> moving it to some other storage, I think that goes against this principle >>>>> since a JSON file is quite open by definition. Going back to the first >>>>> point, I think a catalog which has such a behavior would *not* be >>>>> considered spec compliant. Another reason this is important is if we think >>>>> about what's healthiest for all users of Iceberg, is to have a healthy >>>>> list >>>>> of options for catalog choices. Storing the metadata JSON in non-open ways >>>>> can make users lives harder for trying out new catalogs since now the >>>>> metadata would be stored in their own way, and the users will have a >>>>> harder >>>>> time accessing their own data. >>>>> >>>>> A last point I'd like to make is I think there's a good discussion to >>>>> be had on how do we validate that a REST Catalog implementation is spec >>>>> compliant. I think that's really beneficial for the ecosystem as a whole. >>>>> Before that, I think first though we'd want to conclude on this topic >>>>> itself. >>>>> >>>>> On Thu, Feb 29, 2024 at 12:29 PM Daniel Weeks < >>>>> daniel.c.we...@gmail.com> wrote: >>>>> >>>>>> > REST spec-compliant catalog does not need to follow the Iceberg >>>>>> spec to commit or store metadata >>>>>> >>>>>> If the REST implementation doesn't follow the Iceberg spec for commit >>>>>> requirements, it's not compliant with the spec. There's no exemption >>>>>> that >>>>>> says if you're using REST you don't need to follow the spec. Why do you >>>>>> think that's the case? >>>>>> >>>>>> I don't believe there's a reason to say that the REST spec needs to >>>>>> enforce the commit requirements either, that's a requirement of the >>>>>> Iceberg >>>>>> spec and still needs to be complied with. >>>>>> >>>>>> -Dan >>>>>> >>>>>> On Thu, Feb 29, 2024 at 12:19 PM Jack Ye <yezhao...@gmail.com> wrote: >>>>>> >>>>>>> > The implementation of the spec can either be compliant or not. >>>>>>> >>>>>>> This is exactly the problem we are talking about right? Just to give >>>>>>> an example, we cannot technically say that tables/views in the Tabular >>>>>>> catalog are Iceberg tables/views, because a REST spec-compliant catalog >>>>>>> does not need to follow the Iceberg spec to commit or store metadata. >>>>>>> Even >>>>>>> if you say it is, there is no way to really prove that, because the REST >>>>>>> spec does not enforce it. >>>>>>> >>>>>>> JB, what do you mean by participating on the Catalog RFC? Is there >>>>>>> already an ongoing RFC? >>>>>>> >>>>>>> -Jack >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 29, 2024 at 12:08 PM Jean-Baptiste Onofré < >>>>>>> j...@nanthrax.net> wrote: >>>>>>> >>>>>>>> Hi Dan, >>>>>>>> >>>>>>>> I agree with your statement about REST Spec is not an implement but >>>>>>>> I strongly disagree with your statement "impl of the spec can either be >>>>>>>> compliant or not". >>>>>>>> >>>>>>>> The REST Catalog spec impl should be consistent with the REST Spec. >>>>>>>> That's why a reference implementation in Iceberg would be a must, with >>>>>>>> a >>>>>>>> TCK. >>>>>>>> >>>>>>>> The REST Spec should bridge/give access to Table/View metadata. I >>>>>>>> think it would make sense to have a resource to GET the Table/View >>>>>>>> metadata, also supporting PUT to update. >>>>>>>> JSON Schema and eventually JSON RPC could help on some area here >>>>>>>> (compliant with OpenAPI). >>>>>>>> >>>>>>>> In another thread, I propose to work on a Catalog RFC, exactly to >>>>>>>> target this. I think it would make sense to have the REST/Catalog RFC >>>>>>>> as >>>>>>>> the main catalog API, so it has to be both consistent (giving access to >>>>>>>> table/view metadata) and extensible (via OpenAPI Extensions for >>>>>>>> instance). >>>>>>>> >>>>>>>> So, I agree with Jack: the minimum would be to have JSON metadata >>>>>>>> exposed by the REST Spec. >>>>>>>> >>>>>>>> @Jack, short term I'm in favor of your proposal, long term, I >>>>>>>> propose to participate on the Catalog RFC (REST Spec). WDYT ? >>>>>>>> >>>>>>>> Thanks ! >>>>>>>> Regards >>>>>>>> JB >>>>>>>> >>>>>>>> >>>>>>>> Le jeu. 29 févr. 2024 à 20:47, Daniel Weeks < >>>>>>>> daniel.c.we...@gmail.com> a écrit : >>>>>>>> >>>>>>>>> Hey Jack, >>>>>>>>> >>>>>>>>> I'm not sure I agree with the framing of this argument. The REST >>>>>>>>> Spec defines a protocol, not an implementation. >>>>>>>>> >>>>>>>>> The implementation of the spec can either be compliant or not. So >>>>>>>>> a REST Implementation that adheres to all the requirements (atomic >>>>>>>>> location >>>>>>>>> swap, json representation, etc.), would be compliant. There's no >>>>>>>>> requirement around who performs these operations and with REST, that >>>>>>>>> is >>>>>>>>> delegated to the server. The optional metadata location doesn't mean >>>>>>>>> that >>>>>>>>> there isn't a metadata location, just that it may not be exposed >>>>>>>>> directly >>>>>>>>> in the response. >>>>>>>>> >>>>>>>>> Therefore, an implementation where you just store the table >>>>>>>>> metadata in a Glue Table object, would not be compliant, currently. >>>>>>>>> >>>>>>>>> We've periodically discussed removing the storage requirement and >>>>>>>>> I think there's a path forward to do that and would agree that >>>>>>>>> standardizing on REST, but I wouldn't say the justification for >>>>>>>>> making this >>>>>>>>> push is that REST is not compliant so we can just ignore the table >>>>>>>>> spec >>>>>>>>> requirements. >>>>>>>>> >>>>>>>>> There are a few more things to consider, which is that not >>>>>>>>> everything can use REST currently and making a hard cut away from file >>>>>>>>> based metadata could bifurcate access to Iceberg data. There are also >>>>>>>>> aspects to the spec that reference the metadata paths (like metadata >>>>>>>>> log, >>>>>>>>> though it's optional), but would likely need to be addressed. >>>>>>>>> >>>>>>>>> -Dan >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Feb 29, 2024 at 11:13 AM Jack Ye <yezhao...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi everyone, >>>>>>>>>> >>>>>>>>>> Just want to pull this specific topic out of the materialized >>>>>>>>>> view discussion thread. I noticed this during the MV discussion, and >>>>>>>>>> I >>>>>>>>>> think it is important to clarify this not just for the MV topic, but >>>>>>>>>> also >>>>>>>>>> for the ongoing discussion to consolidate all the different catalogs. >>>>>>>>>> >>>>>>>>>> *How the table/view spec defines Iceberg table/view* >>>>>>>>>> >>>>>>>>>> If we look into the table/view spec, the optimistic concurrency >>>>>>>>>> section <https://iceberg.apache.org/spec/#optimistic-concurrency> >>>>>>>>>> requires the existence of a metadata file, and the atomic swap of the >>>>>>>>>> metadata file ensures serializable isolation. This implies 2 things: >>>>>>>>>> 1. the metadata file in a storage that holds the information >>>>>>>>>> described in the rest of the spec. >>>>>>>>>> 2. there is an object in a catalog that holds the pointer of the >>>>>>>>>> metadata file. What object and what catalog is implementation >>>>>>>>>> dependent, >>>>>>>>>> but these generalized concepts are always intact. >>>>>>>>>> >>>>>>>>>> The JSON serialization parts of the spec plus the reader >>>>>>>>>> requirements also implies that the metadata file is in JSON format. >>>>>>>>>> >>>>>>>>>> So when we talk about an Iceberg table/view that is compliant >>>>>>>>>> with the spec, it is the combination of all these 5 requirements: >>>>>>>>>> 1. there is an object in the catalog representing this table/view >>>>>>>>>> 2. there is a pointer to a JSON metadata file in the object >>>>>>>>>> 3. the JSON metadata file exists in storage and contains the >>>>>>>>>> table/view metadata content >>>>>>>>>> 4. the metadata content is compliant with the standard described >>>>>>>>>> in the spec >>>>>>>>>> 5. serializable isolation is achieved by atomic swap of the >>>>>>>>>> object pointer >>>>>>>>>> >>>>>>>>>> *How non-REST catalogs are compliant with the table/view spec* >>>>>>>>>> >>>>>>>>>> An implementation of the Iceberg table/view is essentially >>>>>>>>>> specifying: >>>>>>>>>> 1. what is the exact implementation of the catalog, e.g. JDBC, >>>>>>>>>> Hive metastore (HMS), Glue, etc. >>>>>>>>>> 2. what is the object that represents a table, e.g. a row in the >>>>>>>>>> "iceberg_tables" table in JDBC, a Table object in HMS/Glue, etc. >>>>>>>>>> 3. how is the JSON metadata file pointer stored, e.g. a column in >>>>>>>>>> the table's row in JDBC, metadata_location key in the Table's >>>>>>>>>> parameter map >>>>>>>>>> in HMS/Glue, etc. >>>>>>>>>> 4. how the atomic swap is implemented, e.g. SQL atomic update in >>>>>>>>>> JDBC, conditional parameter update in HMS, conditional version >>>>>>>>>> update in >>>>>>>>>> Glue, etc. >>>>>>>>>> >>>>>>>>>> *How the REST spec is NOT compliant with the table/view spec* >>>>>>>>>> >>>>>>>>>> The REST spec technically does not match the following table/view >>>>>>>>>> spec requirements: >>>>>>>>>> 2. there is a pointer to a JSON metadata file in the object >>>>>>>>>> 3. the JSON metadata file exists in storage and contains the >>>>>>>>>> table/view metadata content >>>>>>>>>> 5. serializable isolation is achieved by atomic swap of the >>>>>>>>>> object pointer >>>>>>>>>> >>>>>>>>>> The key parts in REST spec that are not compliant are: >>>>>>>>>> 1. metadata-location field is optional in LoadTableResponse >>>>>>>>>> <https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L2721-L2728> >>>>>>>>>> 2. pointer swap is not enforced in the UpdateTable >>>>>>>>>> <https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L658> >>>>>>>>>> operation >>>>>>>>>> >>>>>>>>>> Therefore, it opens the door for a REST service to be completely >>>>>>>>>> not dependent on a JSON metadata file, store the Iceberg table/view >>>>>>>>>> metadata not as a file, and achieve much better performance >>>>>>>>>> characteristics >>>>>>>>>> than other catalogs. This technically gives a unique advantage for >>>>>>>>>> REST >>>>>>>>>> catalog adopters that is not there for non-REST catalogs like HMS >>>>>>>>>> and Glue. >>>>>>>>>> >>>>>>>>>> *How can we fix this?* >>>>>>>>>> >>>>>>>>>> I suggest the following: >>>>>>>>>> >>>>>>>>>> Firstly, I think it is good that we try to remove the >>>>>>>>>> requirements of JSON metadata file pointer and atomic pointer swap. >>>>>>>>>> We know >>>>>>>>>> these requirements have perf limitations based on production usage, >>>>>>>>>> especially when the metadata file is large. If that is the >>>>>>>>>> direction, we >>>>>>>>>> should make it official by changing the table/view spec to say that >>>>>>>>>> those >>>>>>>>>> requirements are catalog level implementation details that are no >>>>>>>>>> longer >>>>>>>>>> required. >>>>>>>>>> >>>>>>>>>> Secondly, once we do that, we should declare REST spec as the >>>>>>>>>> official catalog spec to interact with Iceberg tables. Otherwise at >>>>>>>>>> least I >>>>>>>>>> will be very tempted to just break the atomic pointer swap pattern >>>>>>>>>> and >>>>>>>>>> store the entire metadata using the Glue Table object to achieve much >>>>>>>>>> better performance and also Glue native feature integrations, and I >>>>>>>>>> think >>>>>>>>>> other players will be equally motivated to do something similar. >>>>>>>>>> That will >>>>>>>>>> lead to even more chaos in the Iceberg catalog space. >>>>>>>>>> >>>>>>>>>> With REST spec as the official catalog spec, we can actually >>>>>>>>>> support non-REST catalogs by using the HTTP execution chain handler. >>>>>>>>>> Dan >>>>>>>>>> has already done a prototype here >>>>>>>>>> <https://github.com/apache/iceberg/commit/619127ff69f89e43a1edef2ea94c3dd439396a8d#diff-869264a83ba9ca657e7defefaa16ad196b0de9fce6c87f97533db77f29e44762> >>>>>>>>>> that is based on this discussion >>>>>>>>>> <https://github.com/apache/iceberg/pull/8091#issuecomment-1647189146> >>>>>>>>>> in the past about using AWS Lambda as an alternative HTTP client for >>>>>>>>>> REST >>>>>>>>>> catalog. The same approach can be used to talk to HMS/Glue/JDBC/... >>>>>>>>>> while >>>>>>>>>> users will only interact with the RESTCatalog as the entry point. >>>>>>>>>> >>>>>>>>>> I think this can provide a good path forward overall for the >>>>>>>>>> catalog consolidation story, interested to know what others think. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Jack Ye >>>>>>>>>> >>>>>>>>>>
