Hi everyone, This is a great discussion. I definitely agree that expanding ALTER VIEW support is the best method to allow for defining views across engines conveniently.
A common translation layer is the only way to ensure that all representations are consistent. Any other method requires an engine capable of parsing SQL from multiple dialects. Given that constraint, I think it's fine to place the impetus on the user to ensure that their representations match when they run additional ALTER VIEW commands to add new representations. - Alex Stephen On Tue, Sep 8, 2026 at 12:13 PM Daniel Weeks <[email protected]> wrote: > Thanks Ajantha for digging up the thread on this discussion. > > I think the important takeaway here is understanding what Iceberg is > trying to solve with views, and what it isn't. > > We know that engines have different sql dialects and semantics. Iceberg > views enable exposing metadata in a way that multiple engines can consume > in their native dialect (the representation, not the semantics). > > However, there is no intention to automate the translation or ensure > consistency across representations. This is an important detail that the > user must handle. Translation frameworks or manual verification can be > used, but achieving consistent behavior is complicated and outside the > project's scope. > > If we think we need more convenient ways to update the syntax, that's > worth exploring. However, coordinating representations and verifying > correctness remains the responsibility of the view creator (via API or > natively through the engine). It's easy to get this wrong, but the utility > of views outweighs the burden we place on users. > > I see 'ALTER VIEW' support as a way to make defining the views more > convenient (than, say, using PyIceberg which can't validate the syntax of > other engines). It allows the native engine to parse, resolve all > references, and ensure the view can be executed, but notably does nothing > to ensure the results are consistent with any other representation. > > -Dan > > > On Mon, Sep 7, 2026 at 11:47 PM Ajantha Bhat <[email protected]> > wrote: > >> This discussion feels very similar to the earlier thread that I opened on >> view interoperability a long time ago. >> >> One of the takeaways there was that we wanted to avoid each engine >> manually implementing or maintaining dialect translations. Instead, the >> preference was to have a centralized translation layer (at the time we >> discussed exposing that through PyIceberg) rather than pushing that >> responsibility into every engine. Sharing the previous discussion for >> context, as it covers many of the same trade-offs: >> https://lists.apache.org/thread/k6szpr5smyrh37sy563xpgjor4g6pr81 >> <https://lists.apache.org/thread/k6szpr5smyrh37sy563xpgjor4g6pr81?utm_source=chatgpt.com> >> >> - Ajantha >> >> On Tue, Sep 8, 2026 at 12:09 PM Péter Váry <[email protected]> >> wrote: >> >>> In my opinion, the ideal experience is for users to define both SQL >>> statements when creating the view. This avoids any period during which the >>> view is only partially defined and allows the entire workflow to be handled >>> through a single authoring tool. >>> >>> That said, the Iceberg Java API is available, and engines are free to >>> choose which capabilities they support and how they expose them to users. >>> >>> Talat Uyarer <[email protected]> ezt írta (időpont: 2026. szept. 7., H, >>> 17:18): >>> >>>> Hi Péter, >>>> >>>> I think a SQL-only workflow is closer than it looks. a Spark user >>>> doesn't need Spark to parse the Flink SQL, they need to read it, and >>>> showing another engine's SQL as text is engine-agnostic. What's missing is >>>> presentation, not translation. >>>> >>>> The workflow I'd propose: >>>> >>>> 1. Flink: CREATE VIEW v AS SELECT ... — version 1, with a `flink` >>>> representation. >>>> 2. Spark user: DESCRIBE EXTENDED VIEW— engines should display *all* >>>> representations with their dialect labels. The strings are already in the >>>> metadata, so this is purely a display convention, no spec change. This >>>> answers "how can the Spark user know the Flink SQL". >>>> 3. The user reads the Flink SQL and authors the Spark equivalent. >>>> 4. Spark: ALTER VIEW v AS SELECT ... — upserts the `spark` >>>> representation and carries the `flink` one forward unchanged. Issuing that >>>> statement *is* the consistency assertion — the "user authority" you >>>> described, made concrete. The version summary already records engine >>>> name/version, so we get per-version provenance for free. >>>> 5. On read, each engine executes exactly its own dialect. >>>> >>>> The rule that makes this flag-free is, I think, just a crisper version >>>> of what Dan >>>> said: >>>> >>>> ALTER VIEW ... AS = update/add my engine's representation, >>>> preserve the rest >>>> CREATE OR REPLACE VIEW = redefine the view; only my representation >>>> remains >>>> >>>> Two intents, two existing syntaxes, no properties, no new grammar. >>>> >>>> When someone genuinely changes the view's meaning with CREATE OR >>>> REPLACE, the other representations are gone, the next read from another >>>> engine fails fast, and that engine's user re-adds their dialect via steps >>>> 2–4. The failure is the signal, no cross-engine coordination protocol >>>> needed. >>>> >>>> That only holds if engines actually fail fast, which is Alex's fallback >>>> point: today sqlFor() silently returns the "closest" representation, which >>>> is exactly how an engine ends up executing another engine's SQL. That >>>> leniency was defensible while appending a dialect was impossible; with this >>>> workflow it loses its justification. #17874 adds an opt-in >>>> view-dialect-strict to Flink as a first step, and ideally the error message >>>> itself teaches the workflow: "View v has no 'flink' >>>> representation (found: spark). Use SHOW CREATE VIEW to see the existing >>>> SQL and ALTER VIEW to add a flink representation." >>>> >>>> Talat >>>> >>>> >>>> On 2026/09/07 14:38:10 Péter Váry wrote: >>>> > Do I understand correctly that the expected behavior for the user if >>>> they >>>> > want to create a view with Flink and Spark representations is: >>>> > >>>> > 1. CREATE VIEW my_view AS SELECT ... - in Flink >>>> > 2. ALTER VIEW my_view AS SELECT ... - in Spark >>>> > >>>> > How can a Spark user know the Flink SQL the user used to create the >>>> view? >>>> > >>>> > I'm trying to put together a workflow for creating interoperable >>>> views and >>>> > using SQL doesn't seem viable, so as Ryan mentioned the only option is >>>> > Iceberg API for now. >>>> > >>>> > >>>> > Péter Váry <[email protected]> ezt írta (időpont: 2026. >>>> szept. >>>> > 5., Szo, 9:16): >>>> > >>>> > > Thanks Alex for bringing this up. >>>> > > >>>> > > We had a very similar discussion with Guo when he wanted to add a >>>> Flink >>>> > > representation to a view: >>>> https://github.com/apache/iceberg/issues/15296. >>>> > > Talat also has a fresh PR for this issue in Flink: >>>> > > https://github.com/apache/iceberg/pull/17874 >>>> > > >>>> > > My main concern is consistency. How can Flink be sure that the >>>> Flink SQL >>>> > > returns the same results as the already existing Spark SQL. >>>> > > In the end we accept the users authority to state that the Flink >>>> SQL is >>>> > > consistent with the existing Spark SQL. This means we accept that >>>> they are >>>> > > able to parse both, and make decisions. >>>> > > I do think we should allow a user to update one or more of the SQLs >>>> > > without needing extra flags. >>>> > > >>>> > > >>>> > > >>>> > > On Sat, Sep 5, 2026, 04:43 Prashant Singh <[email protected] >>>> > >>>> > > wrote: >>>> > > >>>> > >> Thanks for starting the thread Alex, >>>> > >> >>>> > >> I agree with Dan, an engine should only update / add their own >>>> dialect >>>> > >> and not make guarantees about other engine dialects without >>>> passing through >>>> > >> the other engine's compiler. >>>> > >> I would recommend ALTER too, I know engines such as starrocks have >>>> came >>>> > >> up with grammar [1] to be more specific that one is adding / >>>> modifying the >>>> > >> dialect, given the dialect is of starrocks, that could be an >>>> option : >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> *ALTER VIEW [<catalog>.<database>.]<view_name>( <column_name> [, >>>> > >> <column_name>]){ ADD | MODIFY } DIALECT<query_statement>* >>>> > >> >>>> > >> >>>> > >> [1] >>>> > >> >>>> https://docs.starrocks.io/docs/data_source/catalog/iceberg/DDL/#alter-view-to-update-starrocks-dialect >>>> > >> >>>> > >> Best, >>>> > >> Prashant Singh >>>> > >> >>>> > >> On Fri, Sep 4, 2026 at 4:33 PM Daniel Weeks <[email protected]> >>>> wrote: >>>> > >> >>>> > >>> I feel like if we want to support updating/adding a >>>> representation for a >>>> > >>> specific engine, the right way is to allow 'ALTER VIEW' to update >>>> or add >>>> > >>> the current engine's representation. >>>> > >>> >>>> > >>> I don't think we can expect an engine to add or update other >>>> engines' >>>> > >>> representations because that would potentially require parsing a >>>> different >>>> > >>> SQL syntax than is native to the engine. >>>> > >>> >>>> > >>> For the example you gave where there's an existing >>>> representation in >>>> > >>> Trino, then using `ALTER VIEW . . .` would just set the spark >>>> > >>> representation (as opposed to a `REPLACE` or `CREATE` that would >>>> create a >>>> > >>> new representation with only the spake dialect). >>>> > >>> >>>> > >>> I don't like the idea of adding properties to toggle behavior >>>> though as >>>> > >>> that makes something declarative subject to side-effects of the >>>> > >>> configuration/session. >>>> > >>> >>>> > >>> -Dan >>>> > >>> >>>> > >>> On Fri, Sep 4, 2026 at 3:37 PM Alex Stephen via dev < >>>> > >>> [email protected]> wrote: >>>> > >>> >>>> > >>>> I'm not sure if `CREATE VIEW` would be the correct choice, as >>>> much as >>>> > >>>> `ALTER VIEW`. The user can see that a view exists through `SHOW >>>> > >>>> VIEWS` (which just calls the underlying List Views API), so they >>>> know that >>>> > >>>> they don't need to create a view as much as alter it. >>>> > >>>> >>>> > >>>> The Java implementation >>>> > >>>> < >>>> https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/view/BaseView.java#L114-L130> >>>> surfaces >>>> > >>>> the first representation when a matching one doesn't exist. This >>>> assumes >>>> > >>>> representations are more-or-less compatible between engines, >>>> which isn't >>>> > >>>> the case. >>>> > >>>> >>>> > >>>> I'm concerned about the user journey where a user attempts to >>>> query a >>>> > >>>> view and receives an error message stating they cannot query it >>>> due to a >>>> > >>>> wrong engine or invalid SQL syntax (because their engine is >>>> using the wrong >>>> > >>>> representation). At this point, their only current recourse is >>>> to use the >>>> > >>>> REST endpoints directly to add their new representation. >>>> > >>>> >>>> > >>>> On Fri, Sep 4, 2026 at 3:01 PM Ryan Blue <[email protected]> >>>> wrote: >>>> > >>>> >>>> > >>>>> I think the intent was for this to happen through API >>>> integration. >>>> > >>>>> What is the user interaction that you're trying to achieve? Do >>>> you want >>>> > >>>>> each `CREATE VIEW` call to automatically append a new >>>> representation? >>>> > >>>>> >>>> > >>>>> On Fri, Sep 4, 2026 at 2:32 PM Alex Stephen via dev < >>>> > >>>>> [email protected]> wrote: >>>> > >>>>> >>>> > >>>>>> Hello, >>>> > >>>>>> >>>> > >>>>>> The view spec allows creating a single view with multiple >>>> > >>>>>> engine-specific representations. This allows a single view to >>>> be queried by >>>> > >>>>>> multiple engines, each using its own specific SQL dialect. >>>> > >>>>>> >>>> > >>>>>> In practice, this is difficult to achieve. Users can use the >>>> > >>>>>> `replace.drop-dialect.allowed` view property to allow the >>>> replacing of an >>>> > >>>>>> existing representation. However, there's no way to append a >>>> representation >>>> > >>>>>> (by creating a new ViewVersion with the existing >>>> representations and a new >>>> > >>>>>> one). This means that a direct REST API call is the only way >>>> to create a >>>> > >>>>>> View with multiple representations. >>>> > >>>>>> >>>> > >>>>>> I'd like to propose the creation of a new view property called >>>> > >>>>>> `replace.append-dialect.allowed` that allows users to append a >>>> new >>>> > >>>>>> representation to an existing view. >>>> > >>>>>> >>>> > >>>>>> PR: https://github.com/apache/iceberg/pull/17930 >>>> > >>>>>> >>>> > >>>>>> Please take a look. I'd love to hear from the community >>>> whether there >>>> > >>>>>> is value in this use case. >>>> > >>>>>> >>>> > >>>>>> Thanks! >>>> > >>>>>> >>>> > >>>>>> -- Alex Stephen >>>> > >>>>>> >>>> > >>>>> >>>> > >>>> >>>
