Re: Table Portability Proposal

Indeed, Manu, you're right. However, integrating support for v2 format based on this should be quite simple. Yufei On Wed, Feb 28, 2024 at 1:18 AM Manu Zhang wrote: > Hi Yufei, > > If I'm not mistaken, https://github.com/apache/iceberg/pull/4705 doesn't > support delete files or format v2,

Re: Table Portability Proposal

Hi Yufei, If I'm not mistaken, https://github.com/apache/iceberg/pull/4705 doesn't support delete files or format v2, does it? Manu On Fri, Feb 23, 2024 at 12:41 AM Yufei Gu wrote: > We took a different approach by modifying the metadata. It is a bit heavy > compared to the relative path and

Re: Table Portability Proposal

We took a different approach by modifying the metadata. It is a bit heavy compared to the relative path and s3 access point, but it can be used for any types of storage and any locations. I shared it here, https://github.com/apache/iceberg/pull/4705. Yufei On Tue, Feb 20, 2024 at 6:25 PM Manu

Re: Table Portability Proposal

Hi Jack, Thanks for sharing this idea. Our typical usage of "relative path" is distcp between two HDFS clusters for disaster recovery. It looks to me that by extending this feature, we should always take the authority and scheme from HDFS configurations in that cluster for any path. The downside

Re: Table Portability Proposal

Just to put another alternative solution on the table. In S3FileIO, we implemented the support for S3 access point and bucket alias, which actually accidentally enabled "relative path" if you are just switching bucket name. At read time, you can supply a catalog property "s3.access-points.="

Re: Table Portability Proposal

Hi Ryan Ah ok, I thought that an Iceberg release is "based"/implement a spec (I assumed the opposite is wrong). Thanks for the explanation! Regards JB On Tue, Feb 20, 2024 at 6:04 PM Ryan Blue wrote: > > JB, > > The spec and the reference implementation are released separately so v3 and >

Re: Table Portability Proposal

JB, The spec and the reference implementation are released separately so v3 and 2.0 are independent. There's no requirement that v3 is completed for Iceberg Java 2.0 and the goal of a 2.0 is to have an opportunity to deprecate and remove things so that we don't continue to carry forward and

Re: Table Portability Proposal

Hi Manu Thanks for the reminder. It sounds like a good feature and worth discussing it :). It was my intention to define what we plan to include (or not) in Spec v3 / Iceberg 2.0.0 (I sent a message about that last week). Regards JB On Tue, Feb 20, 2024 at 10:36 AM Manu Zhang wrote: > > Do we

Re: Table Portability Proposal

Do we still want to move forward with this feature? It's on the roadmap for Spec V3 but it hasn't appeared in our discussion for a while. Manu On Sat, Aug 26, 2023 at 2:43 AM Mohit Garg wrote: > hi > > Please review the approach captured here Iceberg Table

Table Portability Proposal

hi Please review the approach captured here Iceberg Table Portability This is a continuation from the previous effort here - Support relative paths and multiple root locations