Hi everyone, Thanks to everyone who has participated in the discussion for FLIP-546. The feedback has been very positive and has helped improve the proposal.
It seems we have reached a good consensus on the direction of the FLIP. If there are no further objections or major concerns, I would like to start the formal *[VOTE]* thread for this proposal tomorrow. Please voice any final thoughts before then. Best regards, Ramin On Tue, Sep 9, 2025 at 3:40 PM Timo Walther <[email protected]> wrote: > Hi Ramin, > > thanks for the FLIP. Given that both CREATE and ALTER functionality are > already present in the code base, providing CREATE OR ALTER for all our > DDL in the near future makes a lot of sense to improve the declarative > nature of Flink SQL. > > +1 for starting with MATERIALIZED TABLE. And later see whether we can > expand to FUNCTION and others in follow up FLIPs. > > Regards, > Timo > > On 09.09.25 14:05, Ron Liu wrote: > > Hi, Ramin > > > > Sorry for late reply. Thanks for the detailed explanation, I think you > > explained it very well, +1. > > We should keep CREATE OR REPLACE and CREATE OR ALTER with their own clear > > semantics. > > > > Best, > > Ron > > > > David Radley <[email protected]> 于2025年9月9日周二 16:18写道： > > > >> Hi Ramin, > >> Thanks for the feedback - that makes sense, > >> Kind regards, David. > >> > >> From: Ramin Gharib <[email protected]> > >> Date: Tuesday, 9 September 2025 at 08:18 > >> To: [email protected] <[email protected]> > >> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-546: Introduce CREATE OR ALTER > for > >> Materialized Tables > >> > >> Hi Ron, > >> > >> Thanks again for your detailed feedback and for raising the points about > >> CREATE > >> OR REPLACE and the existing FLIP-492. > >> > >> Based on your comments and our discussion, I've updated the FLIP > document > >> to include a more detailed rationale in the "Rejected Alternatives" > >> section. This now covers the semantic differences between REPLACE and > >> ALTER, > >> and includes the research on industry precedents like Snowflake. > >> > >> I am following up to see if my previous email and the recent updates to > the > >> FLIP have addressed the concerns you raised. > >> > >> Of course, this question is open to everyone on the list as well. If > there > >> are any other thoughts or remaining concerns about this proposal, please > >> let me know. > >> > >> Best, > >> > >> Ramin > >> > >> On Fri, Sep 5, 2025 at 5:10 PM Ramin Gharib <[email protected]> > wrote: > >> > >>> Hi David, > >>> > >>> Thanks for the feedback and the good questions about the evolution > >>> scenario. > >>> > >>> To clarify the scope of this FLIP, it's focused on introducing the > CREATE > >>> OR ALTER syntax itself, and not on expanding Flink's underlying schema > >>> evolution capabilities. Every (un)supported scenario described in the > >> FLIP > >>> is taken from the source code of the ALTER command. > >>> > >>> The implementation plan for the new CREATE OR ALTER syntax is to > delegate > >>> directly to the existing ALTER MATERIALIZED TABLE ... AS > >>> <select_statement> logic if the table already exists. In other words, a > >>> set of supported evolutions through CREATE OR ALTER will be exactly > what > >>> the standard ALTER command currently supports. > >>> > >>> So, regarding your specific points: > >>> > >>> 1. > >>> > >>> *Changing nullability:* This would only work if the underlying > ALTER > >>> logic already supports it. > >>> 2. > >>> > >>> *Castable type changes:* As you noted, this would require a data > >>> rewrite and is a significant feature in itself. It would need to be > >>> implemented in the core ALTER command first. Once it is, CREATE OR > >>> ALTER would support it by extension. > >>> > >>> Essentially, this FLIP provides the idempotent entry point, while a > >>> separate, existing component handles the actual evolution logic it > >>> triggers. Expanding those capabilities would be a great topic for > future > >>> FLIPs. > >>> > >>> Kind regards, > >>> > >>> Ramin > >>> > >>> On Fri, Sep 5, 2025 at 3:33 PM David Radley <[email protected]> > >>> wrote: > >>> > >>>> Hi Ramin, > >>>> I like this proposal. > >>>> I wondered what you thought of supporting evolutions like : > >>>> > >>>> 1. > >>>> Changing a column or child object to be nullable? > >>>> 2. > >>>> Allowing column type changes if they were castable? This could > require a > >>>> rewriting of the column data - so may not be feasible. > >>>> 3. > >>>> > >>>> > >>>> WDYT? > >>>> Kind regards, David. > >>>> > >>>> From: Ramin Gharib <[email protected]> > >>>> Date: Friday, 5 September 2025 at 11:27 > >>>> To: [email protected] <[email protected]> > >>>> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-546: Introduce CREATE OR ALTER > >> for > >>>> Materialized Tables > >>>> > >>>> Hi Ron, > >>>> > >>>> Thanks again for the excellent feedback and for bringing FLIP-492 [1] > >> into > >>>> the discussion. I want to address your points on CREATE OR REPLACE vs. > >>>> CREATE > >>>> OR ALTER and the SQL standard. > >>>> > >>>> *1. Semantics of REPLACE vs. ALTER* > >>>> > >>>> You’re right that FLIP-492 [1] proposed CREATE OR REPLACE. However, I > >>>> believe the semantics of ALTER are a much better fit for an object as > >>>> complex as a Materialized Table, which has a running job, state, and > >>>> physical data associated with it. > >>>> > >>>> Interestingly, this exact distinction is recognized by other modern > data > >>>> platforms. For example, *Snowflake supports both CREATE OR REPLACE > TABLE > >>>> and CREATE OR ALTER TABLE as distinct commands with different use > cases > >>>> [2].* > >>>> > >>>> - > >>>> > >>>> Their documentation for CREATE OR ALTER TABLE explicitly states it > >>>> *"creates > >>>> a table if it doesn’t exist, or alters it according to the table > >>>> definition... existing data in the table is preserved when > >> possible."* > >>>> This "modify in place" semantic is precisely what this FLIP aims > to > >>>> achieve. > >>>> - > >>>> > >>>> Conversely, their docs describe CREATE OR REPLACE as *"the > equivalent > >>>> of > >>>> using DROP TABLE on the existing table and then creating a new > >>>> table..."* > >>>> This "drop and recreate" semantics is a destructive action that is > >> not > >>>> suitable for a stateful object like a Flink Materialized Table, > where > >>>> the > >>>> goal is to evolve the pipeline, not destroy it. > >>>> > >>>> This is particularly relevant for Flink, as we already have an ALTER > >>>> MATERIALIZED TABLE ... AS <select_statement> command that defines the > >>>> "modify in-place" behavior. My proposal is that the ALTER path of > CREATE > >>>> OR > >>>> ALTER should be implemented by delegating directly to this existing > >> logic. > >>>> This ensures behavioral consistency. Using REPLACE would introduce a > >>>> conflict that doesn't align with our current implementation. > >>>> > >>>> *2. Forward-Looking Resilience* > >>>> > >>>> The ALTER semantic is also more forward-looking. As Flink's evolution > >>>> capabilities become more sophisticated—potentially including complex > >>>> reprocessing strategies that retain historical data—the concept of > >>>> 'altering' a pipeline is more fitting than 'replacing' it. REPLACE > >>>> suggests > >>>> a simple, destructive action, whereas ALTER provides a more resilient > >>>> foundation for nuanced, state-preserving modifications in the future. > >>>> > >>>> *3. The SQL Standard* > >>>> > >>>> You've raised an important point about the SQL standard. A review > shows > >>>> that neither CREATE OR REPLACE nor CREATE OR ALTER are part of the > >> formal > >>>> ANSI/ISO SQL standard*.* They are both widely adopted, vendor-specific > >>>> extensions. > >>>> > >>>> - > >>>> > >>>> *CREATE OR REPLACE* was popularized by systems like PostgreSQL > (for > >>>> views/functions) [3] and Oracle [4]. > >>>> - > >>>> > >>>> *CREATE OR ALTER* was popularized by Microsoft SQL Server [5] and > is > >>>> now > >>>> a key feature in Snowflake [6]. > >>>> > >>>> Since neither is formally "standard," our choice should be guided by > >> which > >>>> one provides the most clarity and consistency for Flink users. The > fact > >>>> that a major platform like Snowflake has implemented both as distinct > >>>> commands reinforces the idea that the industry sees value in their > >>>> different semantics. > >>>> > >>>> *4. Preventing User Confusion* > >>>> > >>>> I believe CREATE OR ALTER will actually be *less confusing* for Flink > >>>> users > >>>> precisely because it maps directly to the existing commands they > already > >>>> know. A user familiar with ALTER MATERIALIZED TABLE ... AS will > >>>> immediately > >>>> understand what the ALTER part of the new command does. > >>>> > >>>> Given that the implementation for FLIP-492 [1] is not yet complete ( > >>>> FLINK-36995 [7] is still open), now is a good time to choose (or > define) > >>>> the syntax with the clearest and safest semantics before any code is > >>>> merged. > >>>> > >>>> In summary, I believe CREATE OR ALTER provides clearer, > non-destructive > >>>> semantics that are more consistent with Flink's existing DDL for > >>>> Materialized Tables, making it the superior choice for both current > >>>> functionality and future resilience. > >>>> > >>>> Thanks again for the great discussion points. I look forward to > hearing > >>>> your thoughts and those of the wider community. > >>>> > >>>> [1] > >>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-492%3A+Support+Query+Modifications+for+Materialized+Tables > >>>> [2] > >>>> > >> > https://docs.snowflake.com/en/sql-reference/sql/create-table#usage-notes > >>>> [3] https://www.postgresql.org/docs/current/sql-createview.html > >>>> [4] > https://docs.oracle.com/cd/E17952_01/mysql-5.7-en/create-view.html > >>>> [5] > >>>> > >>>> > >> > https://learn.microsoft.com/en-us/sql/t-sql/statements/create-view-transact-sql?view=sql-server-ver17#or-alter > >>>> [6] > >>>> > >>>> > >> > https://docs.snowflake.com/en/sql-reference/sql/create-table#label-create-or-alter-table-syntax > >>>> [7] https://issues.apache.org/jira/browse/FLINK-36995 > >>>> > >>>> Best, > >>>> > >>>> Ramin > >>>> > >>>> On Thu, Sep 4, 2025 at 1:54 PM Ron Liu <[email protected]> wrote: > >>>> > >>>>> Hi, Ramin > >>>>> > >>>>> In FLIP-492[1], we introduced the `CREATE OR REPLACE MATERIALIZED > >> TABLE` > >>>>> syntax to support modifying materialized tables. Can we extend this > >>>> syntax > >>>>> to achieve the functionality you need, such as introducing clause > >>>>> parameters to determine whether to replace the entire table or just > >>>> some of > >>>>> its attributes? > >>>>> > >>>>> Regarding the `CREATE OR ALTER TABLE` syntax, I couldn't find it in > >> the > >>>> SQL > >>>>> standard. I'm concerned that this new syntax might confuse users and > >> add > >>>>> new usage costs. What do you think? > >>>>> > >>>>> 1. > >>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-492%3A+Support+Query+Modifications+for+Materialized+Tables > >>>>> > >>>>> Best, > >>>>> Ron > >>>>> > >>>>> Ramin Gharib <[email protected]> 于2025年9月4日周四 15:04写道： > >>>>> > >>>>>> Good morning, Mate, > >>>>>> Thanks for the feedback! When I wrote this, the FLIP was not yet > >>>> closed! > >>>>> I > >>>>>> have added the DISTRIBUTED BY/INTO clause to the FLIP. > >>>>>> > >>>>>> Cheers, > >>>>>> > >>>>>> Ramin > >>>>>> > >>>>>> On Wed, Sep 3, 2025 at 7:58 PM Mate Czagany <[email protected]> > >>>> wrote: > >>>>>> > >>>>>>> Hi Ramin, > >>>>>>> > >>>>>>> Thank you for the proposal, I think this new command makes perfect > >>>>> sense > >>>>>>> and has the potential to improve IaC pipelines. > >>>>>>> > >>>>>>> It's missing from the SQL syntax, but are there any plans to make > >>>> this > >>>>>> new > >>>>>>> command also support the DISTRIBUTED BY/INTO functionality of > >>>> FLIP-542 > >>>>>> [1] > >>>>>>> that was just voted to be implemented yesterday [2]? > >>>>>>> > >>>>>>> Best Regards > >>>>>>> Mate > >>>>>>> > >>>>>>> [1] > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-542%3A+Make+materialized+table+DDL+consistent+with+regular+tables > >>>>>>> [2] > >>>> https://lists.apache.org/thread/40xyjjvklt65tw21o2pmk8r3srprxs5q > >>>>>>> > >>>>>>> On Tue, Sep 2, 2025 at 3:14 PM Ramin Gharib < > >> [email protected]> > >>>>>> wrote: > >>>>>>> > >>>>>>>> Hi everyone, > >>>>>>>> > >>>>>>>> I want to start a discussion on a new Flink Improvement > >> Proposal, > >>>>>>> FLIP-546: > >>>>>>>> Introduce CREATE OR ALTER for Materialized Tables [1]. > >>>>>>>> > >>>>>>>> The introduction of Materialized Tables in FLIP-435 [2] was a > >>>>>> significant > >>>>>>>> step toward simplifying data pipelines. However, as we integrate > >>>> them > >>>>>>> into > >>>>>>>> production environments managed by automation, a key challenge > >>>>> arises. > >>>>>>>> Modern data operations rely on declarative tools (like dbt, > >>>>> Terraform, > >>>>>>>> CI/CD pipelines) that require idempotent commands to manage the > >>>>>> lifecycle > >>>>>>>> of data assets. > >>>>>>>> > >>>>>>>> The current CREATE MATERIALIZED TABLE syntax is not idempotent > >> and > >>>>>> fails > >>>>>>> if > >>>>>>>> the table already exists. This forces developers and tool > >>>> creators to > >>>>>>>> implement complex and risky CREATE IF NOT EXISTS or DROP...IF > >>>> EXISTS > >>>>>>> logic > >>>>>>>> can lead to data loss and complicate the development of reliable > >>>> data > >>>>>>>> management tools. > >>>>>>>> > >>>>>>>> This FLIP proposes introducing a CREATE OR ALTER MATERIALIZED > >>>> TABLE > >>>>>>> command > >>>>>>>> to provide a native, idempotent way to manage materialized > >> tables, > >>>>>>> aligning > >>>>>>>> Flink with best practices for declarative, > >> infrastructure-as-code > >>>>>>>> deployments. > >>>>>>>> > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/0wnXFg > >>>>>>>> < > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-546%3A+Introduce+CREATE+OR+ALTER+for+Materialized+Tables > >>>>>>>>> > >>>>>>>> > >>>>>>>> [2] https://cwiki.apache.org/confluence/x/HYySEQ > >>>>>>>> > >>>>>>>> Best Regards, > >>>>>>>> > >>>>>>>> Ramin Gharib > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> On Wed, Sep 3, 2025 at 7:58 PM Mate Czagany <[email protected]> > >>>> wrote: > >>>>>> > >>>>>>> Hi Ramin, > >>>>>>> > >>>>>>> Thank you for the proposal, I think this new command makes perfect > >>>>> sense > >>>>>>> and has the potential to improve IaC pipelines. > >>>>>>> > >>>>>>> It's missing from the SQL syntax, but are there any plans to make > >>>> this > >>>>>> new > >>>>>>> command also support the DISTRIBUTED BY/INTO functionality of > >>>> FLIP-542 > >>>>>> [1] > >>>>>>> that was just voted to be implemented yesterday [2]? > >>>>>>> > >>>>>>> Best Regards > >>>>>>> Mate > >>>>>>> > >>>>>>> [1] > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-542%3A+Make+materialized+table+DDL+consistent+with+regular+tables > >>>>>>> [2] > >>>> https://lists.apache.org/thread/40xyjjvklt65tw21o2pmk8r3srprxs5q > >>>>>>> > >>>>>>> On Tue, Sep 2, 2025 at 3:14 PM Ramin Gharib < > >> [email protected]> > >>>>>> wrote: > >>>>>>> > >>>>>>>> Hi everyone, > >>>>>>>> > >>>>>>>> I want to start a discussion on a new Flink Improvement > >> Proposal, > >>>>>>> FLIP-546: > >>>>>>>> Introduce CREATE OR ALTER for Materialized Tables [1]. > >>>>>>>> > >>>>>>>> The introduction of Materialized Tables in FLIP-435 [2] was a > >>>>>> significant > >>>>>>>> step toward simplifying data pipelines. However, as we integrate > >>>> them > >>>>>>> into > >>>>>>>> production environments managed by automation, a key challenge > >>>>> arises. > >>>>>>>> Modern data operations rely on declarative tools (like dbt, > >>>>> Terraform, > >>>>>>>> CI/CD pipelines) that require idempotent commands to manage the > >>>>>> lifecycle > >>>>>>>> of data assets. > >>>>>>>> > >>>>>>>> The current CREATE MATERIALIZED TABLE syntax is not idempotent > >> and > >>>>>> fails > >>>>>>> if > >>>>>>>> the table already exists. This forces developers and tool > >>>> creators to > >>>>>>>> implement complex and risky CREATE IF NOT EXISTS or DROP...IF > >>>> EXISTS > >>>>>>> logic > >>>>>>>> can lead to data loss and complicate the development of reliable > >>>> data > >>>>>>>> management tools. > >>>>>>>> > >>>>>>>> This FLIP proposes introducing a CREATE OR ALTER MATERIALIZED > >>>> TABLE > >>>>>>> command > >>>>>>>> to provide a native, idempotent way to manage materialized > >> tables, > >>>>>>> aligning > >>>>>>>> Flink with best practices for declarative, > >> infrastructure-as-code > >>>>>>>> deployments. > >>>>>>>> > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/0wnXFg > >>>>>>>> < > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-546%3A+Introduce+CREATE+OR+ALTER+for+Materialized+Tables > >>>>>>>>> > >>>>>>>> > >>>>>>>> [2] https://cwiki.apache.org/confluence/x/HYySEQ > >>>>>>>> > >>>>>>>> Best Regards, > >>>>>>>> > >>>>>>>> Ramin Gharib > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>>> Unless otherwise stated above: > >>>> > >>>> IBM United Kingdom Limited > >>>> Registered in England and Wales with number 741598 > >>>> Registered office: Building C, IBM Hursley Office, Hursley Park Road, > >>>> Winchester, Hampshire SO21 2JN > >>>> > >>> > >> > >> Unless otherwise stated above: > >> > >> IBM United Kingdom Limited > >> Registered in England and Wales with number 741598 > >> Registered office: Building C, IBM Hursley Office, Hursley Park Road, > >> Winchester, Hampshire SO21 2JN > >> > > > >
