I would vote to keep a log in the doc with open questions, and keep the doc updated with open questions as they arise/get resolved.
On Tue, Feb 20, 2024 at 11:37 AM Jack Ye <yezhao...@gmail.com> wrote: > Thanks for the response from everyone! > > Before proceeding further, I see a few people referring back to the > current design from Jan. I specifically raised this thread based on the > information in the doc and a few latest discussions we had there. Because > there are many threads in the doc, and each thread points further to other > discussion threads in the same doc or other doc, it is now quite hard to > follow and continue discussing all different topics there. > > I hope we can make incremental consensus of the questions in the doc > through devlist, because it provides more visibility, and also a single > thread instead of multiple threads going on at the same time. If we think > this format is not effective, I propose that we create a new mv channel in > Iceberg Slack workspace, and people interested can join and discuss all > these points directly. What do we think? > > Best, > Jack Ye > > > > On Mon, Feb 19, 2024 at 6:03 PM Szehon Ho <szehon.apa...@gmail.com> wrote: > >> Hi, >> >> Great to see more discussion on the MV spec. Actually, Jan's document >> "Iceberg >> Materialized View Spec" >> <https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A> >> has >> been organized , with a "Design Questions" section to track these debates, >> and it would be nice to centralize the debates there, as Micah mentions. >> >> For Dan's question, I think this debate was tracked in "Design Question >> 3: Should the storage table be registered in the catalog?". I think the >> general idea there was to not expose it directly via Catalog as it is then >> exposed to user modification. If the engine wants to access anything about >> the storage table (including audit and storage), it is of course there via >> the storage table pointer. I think Walaa's point is also good, we could >> expose it as we expose metadata tables, but I am still not sure if there is >> still some use-cases of engine access not covered? >> >> It is true that for Jack's initial question (Do we really want to go with >> the MV = view + storage table design approach for Iceberg MV?), >> unfortunately we did not capture it as a "Design Question" in Jan's doc, as >> it was an implicit assumption of 'yes', because it is the choice of Hive, >> Trino, and other engines , as others have pointed out. >> >> Jack's point about potential evolution of MV (like to add partitioning) >> is an interesting one, but definitely hard to grasp. I think it makes >> sense to add this as a separate Design Question in the doc, and add the >> options. This will allow us to flesh out this alternative >> option(s). Maybe Micah's point about modifying existing proposal to >> 'embed' the required table metadata fields in the existing view metadata, >> is one middle ground option. Or we add a totally new MV object spec for >> MV, separate than existing View spec? >> >> Also , as Jack pointed out, it may make sense to have the REST / Catalog >> API proposal in the doc to educate the above decision. >> >> Thanks >> Szehon >> >> On Mon, Feb 19, 2024 at 4:08 PM Walaa Eldin Moustafa < >> wa.moust...@gmail.com> wrote: >> >>> I think it would help if we answer the question of whether an MV is a >>> view + storage table (and degree of exposing this underlying >>> implementation) in the context of the user interfacing with those concepts: >>> >>> For the end user, interfacing with the engine APIs (e.g., through SQL), >>> materialized view APIs should be almost the same as regular view APIs >>> (except for operations specific to materialized views like REFRESH command >>> etc). Typically, the end user interacts with the (materialized) view object >>> as a view, and the engine performs the abstraction over the storage table. >>> >>> For the engines interfacing with Iceberg, it sounds the correct >>> abstraction at this layer is indeed view + storage table, and engines could >>> have access to both objects to optimize queries. >>> >>> So in a sense, the engine will ultimately hide most of the >>> storage detail from the end user (except for advanced users who want to >>> explicitly access the storage table with a modifier like >>> "db.view.storageTable" -- and they can only read it), while Iceberg will >>> expose the storage details to the engine catalog to use it in scans if >>> needed. So the storage table is hidden or exposed based on the context/the >>> actual users. From Iceberg point of view (which interacts with the >>> engines), the storage table is exposed. Note that this does not >>> necessarily mean that the storage table is registered in the catalog with >>> its own independent name (e.g., where we can drop the view but keep the >>> storage table and access it from the catalog). Addressing the storage table >>> using a virtual namespace like "db.view.storageTable" sounds like a good >>> middle ground. Anyways, end users should not need to directly access the >>> storage table in most cases. >>> >>> Thanks, >>> Walaa. >>> >>> On Mon, Feb 19, 2024 at 3:38 PM Micah Kornfield <emkornfi...@gmail.com> >>> wrote: >>> >>>> Hi Jack, >>>> >>>> >>>>> In my mind, the first key point we all need to agree upon to move this >>>>> design forward is*: Do we really want to go with the MV = view + >>>>> storage table design approach for Iceberg MV?* >>>> >>>> >>>> I think we want this to the extent that we do not want to redefine the >>>> same concept with different representations/naming to the greatest degree >>>> possible. This is why borrowing the concepts from the view (e.g. multiple >>>> ways of expressing the same view logic in different dialects) and aspects >>>> of the materialized data (e.g. partitioning, ordering) feels most natural. >>>> IIUC your proposal, I think you are saying maybe two modifications to the >>>> existing proposals in the document: >>>> >>>> 1. No separate storage table link, instead embed most of the metadata >>>> of the materialized table into the MV document (the exception seems to be >>>> snapshot history) >>>> 2. For snapshot history, have one unified history specific to the MV. >>>> >>>> This seems fairly reasonable to me and I think I can solve some >>>> challenges with the existing proposal in an elegant way. If this is >>>> correct (or maybe if it isn't quite correct) perhaps you can make >>>> suggestions to the document so all of the trade-offs can be discussed in >>>> one place? >>>> >>>> I think the one thing the current draft of the materialized view >>>> ignores is how to store algebraic summaries (e.g. separate sum and count >>>> for averages, or other sketches), so that new data can be incrementally >>>> incorporated. But representing these structures feels like it potentially >>>> has value beyond just MVs (e.g. it can be a natural way to express summary >>>> statistics in table metadata), so I think it deserves at least a try in >>>> incorporating the concepts in the table specification, so the definitions >>>> can be shared. I was imagining this could come as part of the next >>>> revision of MV specification. >>>> >>>> The MV internal structure could evolve in a way that works more >>>>> efficiently with the reduced scope of functionalities, without relying on >>>>> table to offer the same capabilities. I can at least say that is true >>>>> based >>>>> on my internal knowledge of how Redshift MVs work. >>>> >>>> >>>> I'm not sure I fully understand this point, but it seems the main >>>> question here is what would break if it started to evolve in this >>>> direction. Is it purely additive or do we suspect some elements would need >>>> to be removed? My gut feeling here is the main concerns here are getting >>>> the cardinatities correct (i.e. 1 MV should probably have 0, 1 or more >>>> materialized storage tables associated with it, to support more advanced >>>> algebraic structures listed above, and perhaps a second without them, and >>>> additional metadata to distinguish between these two different modes). >>>> >>>> If after the evaluation, we are confident that the MV = view + storage >>>>> table approach is the right way to go, then we can debate the other >>>>> issues, >>>>> and I think the next issue to reach consensus should be "Should the >>>>> storage >>>>> table be registered in the catalog?". >>>> >>>> >>>> I actually think there are actually more fundamental questions posed: >>>> 1. Should be considering how items should be modelled in the REST API >>>> concurrently with the Iceberg spec, as that potentially impacts design >>>> decision (I think the answer is yes, and we should update the doc with >>>> sketches on new endpoints and operations on the endpoints to ensure things >>>> align). >>>> 2. Going forward should new aspects of Iceberg artifacts rely on the >>>> fact that a catalog is present and we can rely on a naming convention for >>>> looking up other artifacts in a catalog as pointers (I lean yes on this, >>>> but I'm a little bit more ambivalent). >>>> >>>> Thanks, >>>> Micah >>>> >>>> On Mon, Feb 19, 2024 at 12:52 PM Jack Ye <yezhao...@gmail.com> wrote: >>>> >>>>> I suggest we need a step-by-step process to make incremental >>>>> consensus, otherwise we are constantly talking about many different >>>>> debates >>>>> at the same time. >>>>> >>>>> In my mind, the first key point we all need to agree upon to move this >>>>> design forward is*: Do we really want to go with the MV = view + >>>>> storage table design approach for Iceberg MV?* >>>>> >>>>> I think we (at least me) started with this assumption, mostly because >>>>> this is how Trino implements MV, and how Hive tables store MV information >>>>> today. But does it mean we should design it that way in Iceberg? >>>>> >>>>> Now I look back at how we did the view spec design, we could also say >>>>> that we just add a representation field in the table spec to store view, >>>>> and an Iceberg view is just a table with no data but with representations >>>>> defined. But we did not do that. So it feels now quite inconsistent to say >>>>> we want to just add a few fields in the table and view spec to call it an >>>>> Iceberg MV. >>>>> >>>>> If we look into most of the other database systems (e.g. Redshift, >>>>> BigQuery, Snowflake), they never expose such implementation details like >>>>> storage table. Apart from being close-sourced systems, I think it is also >>>>> for good technical reasons. There are many more things that a table needs >>>>> to support, but does not really apply to MV. The MV internal structure >>>>> could evolve in a way that works more efficiently with the reduced scope >>>>> of >>>>> functionalities, without relying on table to offer the same capabilities. >>>>> I >>>>> can at least say that is true based on my internal knowledge of how >>>>> Redshift MVs work. >>>>> >>>>> I think we should fully evaluate both directions, and commit to one >>>>> first before debating more things. >>>>> >>>>> If we have a new and independent Iceberg MV spec, then an Iceberg MV >>>>> is under-the-hood a single object containing all MV information. It has >>>>> its >>>>> own name, snapshots, view representation, etc. I don't believe we will be >>>>> blocked by Trino due to its MV SPIs currently requiring the existence of a >>>>> storage table, as it will just be a different implementation from the >>>>> existing one in Trino-Iceberg. In this direction, I don't think we need to >>>>> have any further debate about pointers, metadata locations, storage table, >>>>> etc. because everything will be new. >>>>> >>>>> If after the evaluation, we are confident that the MV = view + storage >>>>> table approach is the right way to go, then we can debate the other >>>>> issues, >>>>> and I think the next issue to reach consensus should be "Should the >>>>> storage >>>>> table be registered in the catalog?". >>>>> >>>>> What do we think? >>>>> >>>>> -Jack >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Feb 19, 2024 at 11:32 AM Daniel Weeks <dwe...@apache.org> >>>>> wrote: >>>>> >>>>>> Jack, >>>>>> >>>>>> I think we should consider either allowing the storage table to be >>>>>> fully exposed/addressable via the catalog or allow access via namespacing >>>>>> like with metadata tables. E.g. <catalog>.<database>.<table>.<storage>, >>>>>> which would allow for full access to the underlying table. >>>>>> >>>>>> For other engines to interact with the storage table (e.g. to execute >>>>>> the query to materialize the table), it may be necessary that the table >>>>>> is >>>>>> fully addressable. Whether the storage table is returned as part of list >>>>>> operations may be something we leave up to the catalog implementation. >>>>>> >>>>>> I don't think the table should reference a physical location (only a >>>>>> logical reference) since things will be changing behind the view >>>>>> definition >>>>>> and I'm not confident we want to have to update the view representation >>>>>> everytime the storage table is updated. >>>>>> >>>>>> I think there's still some exploration as to whether we need to model >>>>>> this as separate from view endpoints, but there may be enough overlap >>>>>> that >>>>>> it's not necessary to have yet another set of endpoints for materialized >>>>>> views (maybe filter params if you need to distinguish?). >>>>>> >>>>>> -Dan >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Feb 18, 2024 at 6:57 PM Renjie Liu <liurenjie2...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, Jack: >>>>>>> >>>>>>> Thanks for raising this. >>>>>>> >>>>>>> In most database systems, MV, view and table are considered >>>>>>>> independent objects, at least at API level. It is very rare for a >>>>>>>> system to >>>>>>>> support operations like "materializing a logical view" or "upgrading a >>>>>>>> logical view to MV", because view and MV are very different in almost >>>>>>>> every >>>>>>>> aspect of user experience. Extending the existing view or table spec to >>>>>>>> accommodate MV might give us a MV implementation similar to the current >>>>>>>> Trino or Hive views, save us some effort and a few APIs in REST, but it >>>>>>>> binds us to a very specific design of MV, which we might regret in the >>>>>>>> future. >>>>>>> >>>>>>> >>>>>>> When I reviewed the doc, I thought we were discussing the spec of >>>>>>> materialized view, just like the spec of table metadata, but didn't not >>>>>>> the >>>>>>> user facing api. I would definitely agree that we should consider MV as >>>>>>> another kind of database object in user facing api, even though it's >>>>>>> internally modelled as a view + storage table pointer. >>>>>>> >>>>>>> If we want to make the REST experience good for MV, I think we >>>>>>>> should at least consider directly describing the full metadata of the >>>>>>>> storage table in Iceberg view, instead of pointing to a JSON file. >>>>>>> >>>>>>> >>>>>>> Do you mean we need to add components like >>>>>>> `LoadMaterializedViewResponse`, if so, I would +1 for this. >>>>>>> >>>>>>> *Q2: what REST APIs do we expect to use for interactions with MVs?* >>>>>>> >>>>>>> >>>>>>> As I have mentioned above, I think we should consider MV as another >>>>>>> database object, so I think we should add a set of apis specifically >>>>>>> designed for MV, such as `loadMV`, `freshMV`. >>>>>>> >>>>>>> On Sat, Feb 17, 2024 at 11:14 AM Jack Ye <yezhao...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi everyone, >>>>>>>> >>>>>>>> As we are discussing the spec change for materialized view, there >>>>>>>> has been a missing aspect that is technically also related, and might >>>>>>>> affect the MV spec design: *how do we want to add MV support to >>>>>>>> the REST spec?* >>>>>>>> >>>>>>>> I would like to discuss this in a new thread to collect people's >>>>>>>> thoughts. This topic expands to the following 2 sub-questions: >>>>>>>> >>>>>>>> *Q1: how would the MV spec change affect the REST spec?* >>>>>>>> In the current proposal, it looks like we are using a design where >>>>>>>> a MV is modeled as an Iceberg view linking to an Iceberg storage >>>>>>>> table. At >>>>>>>> the same time, we do not want to expose this storage table in the >>>>>>>> catalog, >>>>>>>> thus the Iceberg view has a pointer to only a metadata JSON file of the >>>>>>>> Iceberg storage table. Each MV refresh updates the pointer to a new >>>>>>>> metadata JSON file. >>>>>>>> >>>>>>>> I feel this does not play very well with the direction that REST is >>>>>>>> going. The REST catalog is trying to remove the dependency to the >>>>>>>> metadata >>>>>>>> JSON file. For example, in LoadTableResponse the only required field >>>>>>>> is the >>>>>>>> metadata, and metadata-location is actually optional. >>>>>>>> >>>>>>>> If we want to make the REST experience good for MV, I think we >>>>>>>> should at least consider directly describing the full metadata of the >>>>>>>> storage table in Iceberg view, instead of pointing to a JSON file. >>>>>>>> >>>>>>>> *Q2: what REST APIs do we expect to use for interactions with MVs?* >>>>>>>> So far we have been thinking about amending the view spec to >>>>>>>> accommodate MV. This entails likely having MVs also being handled >>>>>>>> through >>>>>>>> the view APIs in REST spec. >>>>>>>> >>>>>>>> We need to agree with that first in the community, because this has >>>>>>>> various implications, and I am not really sure at this point if it is >>>>>>>> the >>>>>>>> best way to go. >>>>>>>> >>>>>>>> If MV interactions are through the view APIs, the view APIs need to >>>>>>>> be updated to accommodate MV constructs that are not really related to >>>>>>>> logical views. In fact, most actions performed on MVs are more similar >>>>>>>> to >>>>>>>> actions performed on table rather than view, which involve configuring >>>>>>>> data >>>>>>>> layout, read and write constructs. For example, users might run >>>>>>>> something >>>>>>>> like: >>>>>>>> >>>>>>>> CREATE MATERIALIZED VIEW mv >>>>>>>> PARTITION BY col1 >>>>>>>> CLUSTER BY col2 >>>>>>>> AS ( // some sql ) >>>>>>>> >>>>>>>> then the CreateView API needs to accept partition spec and sort >>>>>>>> order that are completely not relevant for logical views. >>>>>>>> >>>>>>>> When reading a MV, we might even want to have a >>>>>>>> PlanMaterializedView API similar to the PlanTable API we are adding. >>>>>>>> >>>>>>>> *My personal take* >>>>>>>> It feels like we need to reconsider the question of what is the >>>>>>>> best way to model MV in Iceberg. Should it be (1) a view linked to a >>>>>>>> storage table, or (2) a table with a view SQL associated with it, or >>>>>>>> (3) >>>>>>>> it's a completely independent thing. This topic was discussed in the >>>>>>>> past in >>>>>>>> this doc >>>>>>>> <https://docs.google.com/document/d/1QAuy-meSZ6Oy37iPym8sV_n7R2yKZOHunVR-ZWhhZ6Q/edit?pli=1>, >>>>>>>> but at that time we did not have much perspective about aspects like >>>>>>>> REST >>>>>>>> spec, and the view integration was also not fully completed yet. With >>>>>>>> the >>>>>>>> new knowledge, currently I am actually leaning a bit more towards (3). >>>>>>>> >>>>>>>> In most database systems, MV, view and table are considered >>>>>>>> independent objects, at least at API level. It is very rare for a >>>>>>>> system to >>>>>>>> support operations like "materializing a logical view" or "upgrading a >>>>>>>> logical view to MV", because view and MV are very different in almost >>>>>>>> every >>>>>>>> aspect of user experience. Extending the existing view or table spec to >>>>>>>> accommodate MV might give us a MV implementation similar to the current >>>>>>>> Trino or Hive views, save us some effort and a few APIs in REST, but it >>>>>>>> binds us to a very specific design of MV, which we might regret in the >>>>>>>> future. >>>>>>>> >>>>>>>> If we make a new MV spec, it can be made up of fields that already >>>>>>>> exist in the table and view specs, but it is a whole new spec. In this >>>>>>>> way, >>>>>>>> the spec can evolve independently to accommodate MV specific features, >>>>>>>> and >>>>>>>> we can also create MV-related REST endpoints that will evolve >>>>>>>> independently >>>>>>>> from table and view REST APIs. >>>>>>>> >>>>>>>> But on the other side it is definitely associated with more work to >>>>>>>> maintain a new spec, and potentially big refactoring in the codebase to >>>>>>>> make sure operations today that work on table or view can now support >>>>>>>> MV as >>>>>>>> a different object. And it definitely has other problems that I have >>>>>>>> overlooked. I would greatly appreciate any thoughts about this! >>>>>>>> >>>>>>>> Best, >>>>>>>> Jack Ye >>>>>>>> >>>>>>>>
