Hi, Sergey. I was on vacation recently, so sorry for joining this discussion so late.
I’ve carefully reviewed the FLIP, and purely from the perspective of aligning Materialized Table operations with those of a regular Table, I support this proposal in principle. However, in my understanding, Materialized Tables and regular Tables are fundamentally different. A Materialized Table is bound to a specific pipeline that updates its data—this pipeline is generated from the associated query. In contrast, a regular Table isn’t tied to any pipeline; users manually write queries to update its data. Performing an ALTER operation on a regular Table only modifies metadata, whereas performing ALTER on a Materialized Table affects not only metadata but also the underlying data update mechanism. Given this context, I have the following questions: 1. When adding new columns via CREATE or ALTER that are not included in the defining query of the Materialized Table—who is responsible for updating the data in these new columns? I’m unclear about the purpose and use case for adding such columns. Could you provide a concrete example? 2. For ALTER MATERIALIZED TABLE operations like ADD/MODIFY/DROP columns—these changes could cause the pipeline bound to the Materialized Table to fail. What is the exact execution flow for these operations? Could you elaborate on the runtime behavior for each type of operation? Since these actions impact actual data updates—not just metadata—this is a critical concern. In summary, I believe we shouldn’t blindly apply all regular Table operations directly to Materialized Tables. Instead, we should selectively support a subset of operations based on real-world usage scenarios and semantic correctness. What’s your take on this? Best, Ron Sergey Nuyanzin <[email protected]> 于2025年10月8日周三 08:05写道： > Hi Lincoln, > > Thank you for your feedback. > > I guess we already have similar behavior for CTAS, where we could put > more columns than we have for query. > In this case these extra columns should be filled with nulls, and the > query should be rewritten accordingly [1]. > This also means that extra columns should have nullable type (there is > a dedicated validation for this). > It means that for non query columns we have such default values and > query is rewritten taking them into account > > Regarding adding columns with alter, or some other changes like > adding/dropping columns, constraints, distribution > if I understand correctly MaterializedTableManager looking at table > change can decide whether it should recompute materialized table or > not > > Would it make sense? > > [1] > https://github.com/apache/flink/blob/3478ddf08bce49e271f69b922a37ccada6f58688/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/table/SqlCreateTableAsConverter.java#L66-L74 > > > On Tue, Oct 7, 2025 at 4:14 AM Lincoln Lee <[email protected]> wrote: > > > > Thanks Sergey for driving this FLIP, it's a great addition to > materialized > > table! > > > > Since it coincided with China's National Day holiday and everyone is > still > > on > > vacation, we couldn't reply promptly. > > > > I haven't fully reviewed all the content in the FLIP yet, but there's an > > important issue on the ALTER statement: > > > > Unlike a regular CREATE TABLE, Materialized Table derives its schema from > > the defined query, columns are generated based on the query (and, similar > > to a > > materialized view, the underlying data for these columns is tightly > coupled > > to > > the query definition). Therefore, we cannot simply interpret the effect > of > > an > > single `ALTER MATERIALIZED TABLE ADD New_Column` statement. Supporting > this > > likely requires accompanying column default value，and raises > compatibility > > concerns regarding historical data, that is a complex topic we previously > > discussed offline during the design process of FLIP-492. > > > > Also, once Ron is back in the office, he may give a more detailed > comment. > > > > > > Best, > > Lincoln Lee > > > > > > Sergey Nuyanzin <[email protected]> 于2025年10月2日周四 20:15写道： > > > > > Thank you Ramin > > > > > > In case there is no more feedback/objections > > > I would start voting thread next week > > > > > > On Thu, Sep 25, 2025 at 10:43 AM Ramin Gharib <[email protected]> > > > wrote: > > > > > > > > Hi Sergey, > > > > Thanks for driving this! This sounds good to me! +1 > > > > > > > > Cheers, > > > > > > > > Ramin > > > > > > > > On Wed, Sep 24, 2025 at 2:14 PM Sergey Nuyanzin <[email protected] > > > > > wrote: > > > > > > > > > Hi everyone, > > > > > I'd like to start a discussion of FLIP-550 > > > > > Add similar support for CREATE/ALTER operations for MATERIALIZED > > > > > TABLEs as for TABLEs [1]. > > > > > > > > > > This FLIP is another step towards making tables and materialized > > > > > tables more consistent. There was already one improvement in that > > > > > direction like FLIP-542 [2] to add DISTRIBUTION and SHOW > MATERIALIZED > > > > > TABLES support. However there were several more things noticed > > > > > comparing behavior for CREATE and ALTER operations. For instance > right > > > > > now for materialized tables it is impossible to set anything but > table > > > > > constraint while for tables (CREATE TABLE AS) it is possible to > > > > > provide schema definition since FLIP-463 [3], also ALTER operations > > > > > for TABLE is a way more mature than for MATERIALIZED TABLE. This > FLIP > > > > > is about to decrease the difference by enabling more similar > features > > > > > for materialized tables. > > > > > > > > > > Introducing schema definition support for materialized tables will > > > > > provide users with greater control and flexibility and also will > unify > > > > > usage of tables and materialized tables. > > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=387648095 > > > > > > > > > > [2] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-542%3A+Make+materialized+table+DDL+consistent+with+regular+tables > > > > > > > > > > [3] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-463%3A+Schema+Definition+in+CREATE+TABLE+AS+Statement > > > > > > > > > > -- > > > > > Best regards, > > > > > Sergey > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Sergey > > > > > > > -- > Best regards, > Sergey >
