Hi Ramin,
Thanks for the feedback - that makes sense,
Kind regards, David.
From: Ramin Gharib <[email protected]>
Date: Tuesday, 9 September 2025 at 08:18
To: [email protected] <[email protected]>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-546: Introduce CREATE OR ALTER for
Materialized Tables
Hi Ron,
Thanks again for your detailed feedback and for raising the points about
CREATE
OR REPLACE and the existing FLIP-492.
Based on your comments and our discussion, I've updated the FLIP document
to include a more detailed rationale in the "Rejected Alternatives"
section. This now covers the semantic differences between REPLACE and
ALTER,
and includes the research on industry precedents like Snowflake.
I am following up to see if my previous email and the recent updates to the
FLIP have addressed the concerns you raised.
Of course, this question is open to everyone on the list as well. If there
are any other thoughts or remaining concerns about this proposal, please
let me know.
Best,
Ramin
On Fri, Sep 5, 2025 at 5:10 PM Ramin Gharib <[email protected]> wrote:
Hi David,
Thanks for the feedback and the good questions about the evolution
scenario.
To clarify the scope of this FLIP, it's focused on introducing the CREATE
OR ALTER syntax itself, and not on expanding Flink's underlying schema
evolution capabilities. Every (un)supported scenario described in the
FLIP
is taken from the source code of the ALTER command.
The implementation plan for the new CREATE OR ALTER syntax is to delegate
directly to the existing ALTER MATERIALIZED TABLE ... AS
<select_statement> logic if the table already exists. In other words, a
set of supported evolutions through CREATE OR ALTER will be exactly what
the standard ALTER command currently supports.
So, regarding your specific points:
1.
*Changing nullability:* This would only work if the underlying ALTER
logic already supports it.
2.
*Castable type changes:* As you noted, this would require a data
rewrite and is a significant feature in itself. It would need to be
implemented in the core ALTER command first. Once it is, CREATE OR
ALTER would support it by extension.
Essentially, this FLIP provides the idempotent entry point, while a
separate, existing component handles the actual evolution logic it
triggers. Expanding those capabilities would be a great topic for future
FLIPs.
Kind regards,
Ramin
On Fri, Sep 5, 2025 at 3:33 PM David Radley <[email protected]>
wrote:
Hi Ramin,
I like this proposal.
I wondered what you thought of supporting evolutions like :
1.
Changing a column or child object to be nullable?
2.
Allowing column type changes if they were castable? This could require a
rewriting of the column data - so may not be feasible.
3.
WDYT?
Kind regards, David.
From: Ramin Gharib <[email protected]>
Date: Friday, 5 September 2025 at 11:27
To: [email protected] <[email protected]>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-546: Introduce CREATE OR ALTER
for
Materialized Tables
Hi Ron,
Thanks again for the excellent feedback and for bringing FLIP-492 [1]
into
the discussion. I want to address your points on CREATE OR REPLACE vs.
CREATE
OR ALTER and the SQL standard.
*1. Semantics of REPLACE vs. ALTER*
You’re right that FLIP-492 [1] proposed CREATE OR REPLACE. However, I
believe the semantics of ALTER are a much better fit for an object as
complex as a Materialized Table, which has a running job, state, and
physical data associated with it.
Interestingly, this exact distinction is recognized by other modern data
platforms. For example, *Snowflake supports both CREATE OR REPLACE TABLE
and CREATE OR ALTER TABLE as distinct commands with different use cases
[2].*
-
Their documentation for CREATE OR ALTER TABLE explicitly states it
*"creates
a table if it doesn’t exist, or alters it according to the table
definition... existing data in the table is preserved when
possible."*
This "modify in place" semantic is precisely what this FLIP aims to
achieve.
-
Conversely, their docs describe CREATE OR REPLACE as *"the equivalent
of
using DROP TABLE on the existing table and then creating a new
table..."*
This "drop and recreate" semantics is a destructive action that is
not
suitable for a stateful object like a Flink Materialized Table, where
the
goal is to evolve the pipeline, not destroy it.
This is particularly relevant for Flink, as we already have an ALTER
MATERIALIZED TABLE ... AS <select_statement> command that defines the
"modify in-place" behavior. My proposal is that the ALTER path of CREATE
OR
ALTER should be implemented by delegating directly to this existing
logic.
This ensures behavioral consistency. Using REPLACE would introduce a
conflict that doesn't align with our current implementation.
*2. Forward-Looking Resilience*
The ALTER semantic is also more forward-looking. As Flink's evolution
capabilities become more sophisticated—potentially including complex
reprocessing strategies that retain historical data—the concept of
'altering' a pipeline is more fitting than 'replacing' it. REPLACE
suggests
a simple, destructive action, whereas ALTER provides a more resilient
foundation for nuanced, state-preserving modifications in the future.
*3. The SQL Standard*
You've raised an important point about the SQL standard. A review shows
that neither CREATE OR REPLACE nor CREATE OR ALTER are part of the
formal
ANSI/ISO SQL standard*.* They are both widely adopted, vendor-specific
extensions.
-
*CREATE OR REPLACE* was popularized by systems like PostgreSQL (for
views/functions) [3] and Oracle [4].
-
*CREATE OR ALTER* was popularized by Microsoft SQL Server [5] and is
now
a key feature in Snowflake [6].
Since neither is formally "standard," our choice should be guided by
which
one provides the most clarity and consistency for Flink users. The fact
that a major platform like Snowflake has implemented both as distinct
commands reinforces the idea that the industry sees value in their
different semantics.
*4. Preventing User Confusion*
I believe CREATE OR ALTER will actually be *less confusing* for Flink
users
precisely because it maps directly to the existing commands they already
know. A user familiar with ALTER MATERIALIZED TABLE ... AS will
immediately
understand what the ALTER part of the new command does.
Given that the implementation for FLIP-492 [1] is not yet complete (
FLINK-36995 [7] is still open), now is a good time to choose (or define)
the syntax with the clearest and safest semantics before any code is
merged.
In summary, I believe CREATE OR ALTER provides clearer, non-destructive
semantics that are more consistent with Flink's existing DDL for
Materialized Tables, making it the superior choice for both current
functionality and future resilience.
Thanks again for the great discussion points. I look forward to hearing
your thoughts and those of the wider community.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-492%3A+Support+Query+Modifications+for+Materialized+Tables
[2]
https://docs.snowflake.com/en/sql-reference/sql/create-table#usage-notes
[3] https://www.postgresql.org/docs/current/sql-createview.html
[4] https://docs.oracle.com/cd/E17952_01/mysql-5.7-en/create-view.html
[5]
https://learn.microsoft.com/en-us/sql/t-sql/statements/create-view-transact-sql?view=sql-server-ver17#or-alter
[6]
https://docs.snowflake.com/en/sql-reference/sql/create-table#label-create-or-alter-table-syntax
[7] https://issues.apache.org/jira/browse/FLINK-36995
Best,
Ramin
On Thu, Sep 4, 2025 at 1:54 PM Ron Liu <[email protected]> wrote:
Hi, Ramin
In FLIP-492[1], we introduced the `CREATE OR REPLACE MATERIALIZED
TABLE`
syntax to support modifying materialized tables. Can we extend this
syntax
to achieve the functionality you need, such as introducing clause
parameters to determine whether to replace the entire table or just
some of
its attributes?
Regarding the `CREATE OR ALTER TABLE` syntax, I couldn't find it in
the
SQL
standard. I'm concerned that this new syntax might confuse users and
add
new usage costs. What do you think?
1.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-492%3A+Support+Query+Modifications+for+Materialized+Tables
Best,
Ron
Ramin Gharib <[email protected]> 于2025年9月4日周四 15:04写道：
Good morning, Mate,
Thanks for the feedback! When I wrote this, the FLIP was not yet
closed!
I
have added the DISTRIBUTED BY/INTO clause to the FLIP.
Cheers,
Ramin
On Wed, Sep 3, 2025 at 7:58 PM Mate Czagany <[email protected]>
wrote:
Hi Ramin,
Thank you for the proposal, I think this new command makes perfect
sense
and has the potential to improve IaC pipelines.
It's missing from the SQL syntax, but are there any plans to make
this
new
command also support the DISTRIBUTED BY/INTO functionality of
FLIP-542
[1]
that was just voted to be implemented yesterday [2]?
Best Regards
Mate
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-542%3A+Make+materialized+table+DDL+consistent+with+regular+tables
[2]
https://lists.apache.org/thread/40xyjjvklt65tw21o2pmk8r3srprxs5q
On Tue, Sep 2, 2025 at 3:14 PM Ramin Gharib <
[email protected]>
wrote:
Hi everyone,
I want to start a discussion on a new Flink Improvement
Proposal,
FLIP-546:
Introduce CREATE OR ALTER for Materialized Tables [1].
The introduction of Materialized Tables in FLIP-435 [2] was a
significant
step toward simplifying data pipelines. However, as we integrate
them
into
production environments managed by automation, a key challenge
arises.
Modern data operations rely on declarative tools (like dbt,
Terraform,
CI/CD pipelines) that require idempotent commands to manage the
lifecycle
of data assets.
The current CREATE MATERIALIZED TABLE syntax is not idempotent
and
fails
if
the table already exists. This forces developers and tool
creators to
implement complex and risky CREATE IF NOT EXISTS or DROP...IF
EXISTS
logic
can lead to data loss and complicate the development of reliable
data
management tools.
This FLIP proposes introducing a CREATE OR ALTER MATERIALIZED
TABLE
command
to provide a native, idempotent way to manage materialized
tables,
aligning
Flink with best practices for declarative,
infrastructure-as-code
deployments.
[1] https://cwiki.apache.org/confluence/x/0wnXFg
<
https://cwiki.apache.org/confluence/display/FLINK/FLIP-546%3A+Introduce+CREATE+OR+ALTER+for+Materialized+Tables
[2] https://cwiki.apache.org/confluence/x/HYySEQ
Best Regards,
Ramin Gharib
On Wed, Sep 3, 2025 at 7:58 PM Mate Czagany <[email protected]>
wrote:
Hi Ramin,
Thank you for the proposal, I think this new command makes perfect
sense
and has the potential to improve IaC pipelines.
It's missing from the SQL syntax, but are there any plans to make
this
new
command also support the DISTRIBUTED BY/INTO functionality of
FLIP-542
[1]
that was just voted to be implemented yesterday [2]?
Best Regards
Mate
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-542%3A+Make+materialized+table+DDL+consistent+with+regular+tables
[2]
https://lists.apache.org/thread/40xyjjvklt65tw21o2pmk8r3srprxs5q
On Tue, Sep 2, 2025 at 3:14 PM Ramin Gharib <
[email protected]>
wrote:
Hi everyone,
I want to start a discussion on a new Flink Improvement
Proposal,
FLIP-546:
Introduce CREATE OR ALTER for Materialized Tables [1].
The introduction of Materialized Tables in FLIP-435 [2] was a
significant
step toward simplifying data pipelines. However, as we integrate
them
into
production environments managed by automation, a key challenge
arises.
Modern data operations rely on declarative tools (like dbt,
Terraform,
CI/CD pipelines) that require idempotent commands to manage the
lifecycle
of data assets.
The current CREATE MATERIALIZED TABLE syntax is not idempotent
and
fails
if
the table already exists. This forces developers and tool
creators to
implement complex and risky CREATE IF NOT EXISTS or DROP...IF
EXISTS
logic
can lead to data loss and complicate the development of reliable
data
management tools.
This FLIP proposes introducing a CREATE OR ALTER MATERIALIZED
TABLE
command
to provide a native, idempotent way to manage materialized
tables,
aligning
Flink with best practices for declarative,
infrastructure-as-code
deployments.
[1] https://cwiki.apache.org/confluence/x/0wnXFg
<
https://cwiki.apache.org/confluence/display/FLINK/FLIP-546%3A+Introduce+CREATE+OR+ALTER+for+Materialized+Tables
[2] https://cwiki.apache.org/confluence/x/HYySEQ
Best Regards,
Ramin Gharib
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: Building C, IBM Hursley Office, Hursley Park Road,
Winchester, Hampshire SO21 2JN
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: Building C, IBM Hursley Office, Hursley Park Road,
Winchester, Hampshire SO21 2JN
