Good morning, Mate, Thanks for the feedback! When I wrote this, the FLIP was not yet closed! I have added the DISTRIBUTED BY/INTO clause to the FLIP.
Cheers, Ramin On Wed, Sep 3, 2025 at 7:58 PM Mate Czagany <[email protected]> wrote: > Hi Ramin, > > Thank you for the proposal, I think this new command makes perfect sense > and has the potential to improve IaC pipelines. > > It's missing from the SQL syntax, but are there any plans to make this new > command also support the DISTRIBUTED BY/INTO functionality of FLIP-542 [1] > that was just voted to be implemented yesterday [2]? > > Best Regards > Mate > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-542%3A+Make+materialized+table+DDL+consistent+with+regular+tables > [2] https://lists.apache.org/thread/40xyjjvklt65tw21o2pmk8r3srprxs5q > > On Tue, Sep 2, 2025 at 3:14 PM Ramin Gharib <[email protected]> wrote: > > > Hi everyone, > > > > I want to start a discussion on a new Flink Improvement Proposal, > FLIP-546: > > Introduce CREATE OR ALTER for Materialized Tables [1]. > > > > The introduction of Materialized Tables in FLIP-435 [2] was a significant > > step toward simplifying data pipelines. However, as we integrate them > into > > production environments managed by automation, a key challenge arises. > > Modern data operations rely on declarative tools (like dbt, Terraform, > > CI/CD pipelines) that require idempotent commands to manage the lifecycle > > of data assets. > > > > The current CREATE MATERIALIZED TABLE syntax is not idempotent and fails > if > > the table already exists. This forces developers and tool creators to > > implement complex and risky CREATE IF NOT EXISTS or DROP...IF EXISTS > logic > > can lead to data loss and complicate the development of reliable data > > management tools. > > > > This FLIP proposes introducing a CREATE OR ALTER MATERIALIZED TABLE > command > > to provide a native, idempotent way to manage materialized tables, > aligning > > Flink with best practices for declarative, infrastructure-as-code > > deployments. > > > > [1] https://cwiki.apache.org/confluence/x/0wnXFg > > < > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-546%3A+Introduce+CREATE+OR+ALTER+for+Materialized+Tables > > > > > > > [2] https://cwiki.apache.org/confluence/x/HYySEQ > > > > Best Regards, > > > > Ramin Gharib > > On Wed, Sep 3, 2025 at 7:58 PM Mate Czagany <[email protected]> wrote: > Hi Ramin, > > Thank you for the proposal, I think this new command makes perfect sense > and has the potential to improve IaC pipelines. > > It's missing from the SQL syntax, but are there any plans to make this new > command also support the DISTRIBUTED BY/INTO functionality of FLIP-542 [1] > that was just voted to be implemented yesterday [2]? > > Best Regards > Mate > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-542%3A+Make+materialized+table+DDL+consistent+with+regular+tables > [2] https://lists.apache.org/thread/40xyjjvklt65tw21o2pmk8r3srprxs5q > > On Tue, Sep 2, 2025 at 3:14 PM Ramin Gharib <[email protected]> wrote: > > > Hi everyone, > > > > I want to start a discussion on a new Flink Improvement Proposal, > FLIP-546: > > Introduce CREATE OR ALTER for Materialized Tables [1]. > > > > The introduction of Materialized Tables in FLIP-435 [2] was a significant > > step toward simplifying data pipelines. However, as we integrate them > into > > production environments managed by automation, a key challenge arises. > > Modern data operations rely on declarative tools (like dbt, Terraform, > > CI/CD pipelines) that require idempotent commands to manage the lifecycle > > of data assets. > > > > The current CREATE MATERIALIZED TABLE syntax is not idempotent and fails > if > > the table already exists. This forces developers and tool creators to > > implement complex and risky CREATE IF NOT EXISTS or DROP...IF EXISTS > logic > > can lead to data loss and complicate the development of reliable data > > management tools. > > > > This FLIP proposes introducing a CREATE OR ALTER MATERIALIZED TABLE > command > > to provide a native, idempotent way to manage materialized tables, > aligning > > Flink with best practices for declarative, infrastructure-as-code > > deployments. > > > > [1] https://cwiki.apache.org/confluence/x/0wnXFg > > < > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-546%3A+Introduce+CREATE+OR+ALTER+for+Materialized+Tables > > > > > > > [2] https://cwiki.apache.org/confluence/x/HYySEQ > > > > Best Regards, > > > > Ramin Gharib > > >
