Hi folks, It's fantastic to see so much interest, and I'm delighted that the feedback has been largely positive. I’ll address the comments in detail:
*Big Metadata Problem: *First, we are not necessarily reintroducing the scaling issues that HIVE faced, as the metadata itself remains as files in each table’s location. Depending on the server's processing capabilities, there are multiple ways to resolve the metadata problem: 1. *Incremental Updates*: The REST catalog can update metadata incrementally with each commit. This approach can effectively address most scaling issues. An Iceberg table can support a huge amount of data(PB, even EB), but most tables aren't too large, and each update only affects a small percentage of partitions. Incremental updates should be good for them. For the use case of large tables with updates touching most of the partitions, we can rely on the following methods. 2. *Asynchronous Computation*: We’ve already implemented asynchronous updates for partition stats collection, and storing the stats in a parquet file. A similar approach can be introduced for managing files and manifests, if necessary. 3. *Distributed Processing with Fan-Out Workers: *The REST catalog can leverage Trino/Spark for certain distributed processing tasks. Although this method should be considered a last resort due to its complexity and additional dependencies, it is highly effective and still worth doing. By incorporating caching, we can significantly reduce overall file system access. In terms of resource usage, this method is not particularly concerning since it handles metadata only, which is far less resource-intensive compared to data handling. I’m open to exploring other ideas, such as the PrePlan/PlanTable API-based approach, though I'm not entirely familiar with it. I would appreciate it if people could share more insights on this. *Additional metadata endpoints* > I wonder if we can even add an optional endpoint for listing 'removed' snapshots. > Since many proposals need access to at least manifest-lists and manifest files, potentially also data/delete files, does it make sense to bundle all proposals that need this ability? For the above comments from Szehon and Robert, the metadata endpoints aren’t necessarily tied to metadata tables. I proposed it this way for two reasons: 1. *Familiarity*: It’s easier to understand since we are all familiar with metadata tables and their usefulness. 2. *Compatibility*: It’s compatible with existing engines and use cases. Spark, for example, has the most metadata table features, and there has been a consistent demand for similar capabilities in other engines. However, the metadata endpoints could extend beyond the current scope of metadata tables. I’m open to considering this proposal as an umbrella while raising additional metadata endpoint proposals separately. *Response size* For large response size concerns from Gabor, we can combine compression and pagination to mitigate. Thanks a lot for the compression idea from Drew. Yufei On Fri, Jul 5, 2024 at 7:15 AM Vishal Jadhav (BLOOMBERG/ NEW JERSE) < vjad...@bloomberg.net> wrote: > Thinking aloud. > There are some usecases in authorization/authentication realm which could > use this approach. > > From: sn...@snazy.de At: 07/04/24 06:10:53 UTC-4:00 > To: dev@iceberg.apache.org > Subject: Re: [Proposal] REST Spec: Server-side Metadata Tables > > Hi Yufei, > > I think the proposal is very interesting! The direction this and other > proposals are going is IMO the right one. > > Since many proposals need access to at least manifest-lists and manifest > files, potentially also data/delete files, does it make sense to bundle all > proposals that need this ability? > > Robert > On 03.07.24 22:44, Yufei Gu wrote: > > Hi folks, > > I'd like to discuss a new proposal to support server-side metadata tables. > > One of Iceberg's most advantageous features is the ability to inspect a > table using metadata tables. For instance, we can query snapshots just like > we query data rows using the following command: SELECT * FROM > prod.db.table.snapshots; > > With the REST catalog, we can simplify this process further by providing > metadata directly from REST endpoints. Here are several benefits of this > approach: > > - > - Engine Independence: The metadata tables do not rely on a specific > implementation of an engine. The REST server returns the results directly. > For example, the Rust Iceberg does not need to implement its own logic to > query the snapshot table if it connects to a server with this capability. > This reduces the complexity and development effort required for different > clients and engines. > - > - Enabled New Use Cases: A catalog UI or Lakehouse UI can present a > table's metadata (e.g., snapshot/partition list) without relying on an > engine like Trino. This opens up possibilities for lightweight UIs and > tools that can directly interact with the REST endpoints to retrieve and > display metadata. > - > - Enhanced Performance: With server-side caching, the server-side > metadata tables will perform better. Caching reduces the need to repeatedly > compute or retrieve metadata, leading to faster response times and reduced > load on the underlying storage systems. > - > > Here is the proposal in google doc: > https://docs.google.com/document/d/1MVLwyMQtZ-7jewsQ0PuTvtJbpfl4HCoVdbowMqFTmfc/edit?usp=sharing > > Estimated read time: 5 mins > > Would really appreciate any feedback on this topic and proposal! > > > Yufei > > -- > Robert Stupp > @snazy > > >
