I think it might be worth restating perceived requirements and making sure there is alignment on them.
If I am reading correctly, I think the following are perceived requirements: 1. An engine must be able to unambiguously detect that an underlying queried entity has changed or not via metadata to decide if materialized table data can be used. 2. The number of sequential catalog reads an engine needs to make to make use of a materialized table state at read time is minimized. 3. Engines that don't understand a SQL dialect can still use MV information if it is not stale. 4. Table refs (catalog identifiers) should not appear in the materialized table metadata (i.e. state). 5. The view part of the MV definition should not need a new revision for any changes to objects it queries as long as their schemas stay compatible (only state information on the materialized table need to change). In my mind, requirement 1, is the only true requirement. I think this necessitates having UUID + snapshot ID as part of the state information (not necessarily part of the Lineage). I think it also necessitates having a denormalized view of all entities that are inputs into the MV in the state information (a view object might not change but its underlying tables or views could change and that must be detected). Requirements 2 and 5 are somewhat at odds with each other. If information is denormalized (fully expanded) in Lineage, it means if table information is somehow dropped from an intermediate view, one would need to update the view (or make excess calls to the catalog). In my mind, this argues for normalization of the lineage stored on the view (with the cost of potentially 1 additional serial catalog lookup once the state information is retrieved). I think #3 is at odds with #4. I think #3 is more worthwhile, then keeping #4 (and as Jan noted #4 adds complexity). I think the last remaining question is if lineage serves any purpose. I think it is useful for the following reasons: a) When there are no intermediate views queried, it allows for fully parallelized lookup calls to the catalog without having to parse the SQL statement first b) Allows tools that don't need to lookup state information or parse SQL but still navigate MV/view trees. Both of these seem relatively minor, so lineage could perhaps be left out in the first iteration. As it applies to Jan's questions: 1. Should we move the identifiers out of the refresh-state into a new > lineage record that is stored as part of the view metadata? No, I don't think so, I think #5 is a reasonable requirement and I think this violates it. > 2. If yes, should the lineage in the view be fully expanded? No, I think only the state should be fully expanded (for reasons mentioned above, it potentially requires more updates to the view then necessary). > 3. What should be used as an identifier in the lineage to reference > entries in the refresh-state? Catalog identifiers make sense to me. If we agree requirement #3 is not a requirement then it seems like this could also be UUIDs. Thanks, Micah On Thu, Aug 15, 2024 at 7:57 AM Benny Chow <btc...@gmail.com> wrote: > If we go with either UUID or Table Identifier + VersionID/SnapshotId in > the refresh state, then this list is fully expanded already. So, to > validate the freshness of a materialization, the engine doesn't even need > to look at the view lineage. IMO, the view lineage is nice to have but not > a necessary requirement for MVs. The view lineage makes sharing of views > between engines without common SQL dialects possible. > > Benny > > On Thu, Aug 15, 2024 at 12:22 AM Jan Kaul <jank...@mailbox.org.invalid> > wrote: > >> Hi all, >> >> I would like to reemphasize the purpose of the refresh-state for >> materialized views. The purpose is to determine if the precomputed data is >> fresh, stale or invalid. For that the current snapshot-id of every table in >> the query tree has to be fetched from the catalog by using its full >> identifier and ref. Additionally the refresh state stores the snapshot-id >> of the last refresh. >> >> To summarize: *To determine the freshness of the precomputed data we >> require the full identifier + ref and snapshot-id of the last refresh for >> every table in the fully expanded query tree* >> >> This is a requirement from how the catalog works and independent from how >> we design the lineage/refresh state. Additionally we previously agreed that >> we should be able to obtain the full list of identifiers without needing to >> parse the SQL definition. >> >> Now we are having a discussion in how to store and obtain the fully >> expanded list of table identifiers and snapshot-ids. To move the discussion >> forward I think it would be valuable to answer the following 3 questions: >> >> 1. Should we move the identifiers out of the refresh-state into a new >> lineage record that is stored as part of the view metadata? >> >> 2. If yes, should the lineage in the view be fully expanded? >> >> 3. What should be used as an identifier in the lineage to reference >> entries in the refresh-state? >> >> 1. Question: >> >> We already agreed that this would be a good idea because we wouldn't >> introduce the identifier concept to the table metadata. However, looking at >> the complexity that comes with the alternatives, I would like to keep this >> question open. >> >> 2. Question: >> >> I'm against using a not fully expanded lineage in the view struct. To >> recall we require every identifier in the fully expanded query tree to >> determine the freshness. Not storing all identifiers in the lineage would >> mean to recursively call the catalog and expand the query tree at read >> time. This can lead to a large overhead for determining the refresh state >> compared to expanding the query tree once at creation time and then storing >> the fully expanded lineage. >> >> 3. Question: >> >> This depends on Question 2. >> >> For a not fully expanded lineage, the only options would be uuids or >> catalog identifiers. >> >> For a fully expanded lineage the question isn't all that relevant. The >> current design specifies that the lineage is a map from an identifier to an >> id and the refresh-state is a map from such id to a snapshot-id. For this >> to work we don't have to specify which kind of identifier has to be used. >> One query engine could use uuids, the other engine sequence-ids. The >> important assumption we are making is that every id that is used in the >> refresh-state has to be defined in the lineage. >> So the question about using uuids is rather, can the query engine trust >> that the id defined in the lineage is the uuid of the table. >> >> >> Regarding the complexity that comes from introducing the lineage in the >> view I would like to revisit question 1. Introducing the lineage in the >> view metadata opens up the question of when should the lineage be fully >> expanded. We see that we have 3 options: >> >> 1. Not fully expanded lineage -> Expansion at read time >> >> 2. Fully expanded lineage -> Expansion at creation time >> >> 3. No lineage (use identifiers in refresh-state) -> Expansion at refresh >> time >> >> As reading is expected to be the most frequent operation I see option 1 >> as not favorable. As the query engine has to fully expand the query tree >> for a refresh anyway, I see option 3 as the most natural. For a refresh >> operation the query engine must understand the SQL dialects of all views in >> the query tree and therefore is guaranteed to successfully expand the >> lineage. This might not be the case at creation time, which makes option 2 >> less favorable. >> >> As can be seen, I'm in favor of just storing the refresh-state as a map >> from identifier to snapshot-id and not using the lineage. I know that this >> introduces the concept of a catalog identifiers to the table metadata spec, >> but in my opinion it is by far the simplest option. >> >> I'm interested in your opinions. >> >> Best wishes, >> >> Jan >> On 14.08.24 22:24, Walaa Eldin Moustafa wrote: >> >> Thanks Benny. For refs, I am +1 to represent them as UUID + optional ref, >> although we can iterate ohe exact JSON structure (e.g., another option is >> splitting for (UUID) state from (UUID + ref) state into two separate >> higher-level fields). >> >> Generally agree on REFRESH VIEW strategy could be up to the engine, but >> it seems like an area where Iceberg could have an opinion/spec on. I will >> start a separate thread for that. >> >> Thanks, >> Walaa. >> >>
