Hi, Sergey.
Thanks for your quick response.
>>> Probably one of the main use cases here is column name reservation for
the future.
I haven’t seen a concrete real-world business use case for this feature. My
concern is: if the sole motivation is to align syntactically with CTAS
(Create Table As Select), what is its actual value and significance?
>>> Isn't it the same in the case of tables and pipelines bound to them?
I was thinking that since there is MaterializedTableManager, then
based on coming TableChangeOperation
it could decide then how to process such change: for example full
recompute or something more sophisticated
I believe this is not entirely equivalent to a regular table:
- A materialized table consists of multiple components: table metadata,
pipeline, and data. The pipeline is an integral part of the materialized
table and is managed by it. We must ensure the stability and consistency of
all these components.
- Unlike regular tables, the schema and data of a materialized table are
derived from and continuously updated by its defining query. Therefore,
when adding, modifying, or dropping columns, the correct approach should be
to first update the query, and let the query drive the schema change.
This is logically sound and delivers real business value. This capability
is already supported in FLIP-492[1] and FLIP-546 and can be further
extended.
- Allowing column modifications via ALTER MATERIALIZED TABLE
ADD/MODIFY/DROP COLUMN would cause a mismatch between the materialized
table’s query definition and its physical schema, leading to inconsistency
and significantly increasing operational and observability costs. Maybe
also confuse the user.
- Due to the special nature of materialized tables, metadata
modifications must be handled with great caution. We should not allow
arbitrary schema changes—for example, we cannot freely reorder columns or
change column types, and we may not even allow arbitrary column deletions,
as these could break data compatibility. Moreover, if the underlying
physical storage doesn’t support such changes, the pipeline may fail to
run, and the MaterializedTableManager would be unable to handle it. In
such cases, the best solution is for the user to explicitly recreate the
materialized table. We must be cautious with user data—we should not
silently rebuild the physical table or reprocess historical data on the
user’s behalf. Additionally, from a technical perspective, we may currently
lack the capability to perform a full historical backfill in
MaterializedTableManager, so such operations should be explicitly triggered
by the user.
A new question: regarding the operation ALTER MATERIALIZED TABLE MyTable
ADD PARTITION (p1=1, p2='a') WITH ('k1'='v1'), what is its intended
semantics? Is it purely a metadata-only operation, or does it also trigger
a job to refresh data for the new partition? I believe it should be the
latter—what’s your view?
With respect to the operations proposed in the FLIP, I think we can support
those that only affect metadata and do not impact the materialized table’s
query logic or data update behavior. The following operations are
acceptable:
- Support defining watermarks when creating a MATERIALIZED TABLE.
- Support specifying a column_list when creating a MATERIALIZED TABLE.
- Support ALTER MATERIALIZED TABLE to add watermarks, primary keys, or
partitions.
- Support ALTER MATERIALIZED TABLE to drop watermarks, primary keys, or
partitions.
For all other operations that would affect data updates or require query
rewriting, I remain cautious and reserved.
1.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-492%3A+Support+Query+Modifications+for+Materialized+Tables
2.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-546%3A+Introduce+CREATE+OR+ALTER+for+Materialized+Tables
Best,
Ron
Sergey Nuyanzin <[email protected]> 于2025年10月9日周四 18:38写道：
> Hi Ron,
>
> thank you for the feedback
>
> >When adding new columns via CREATE or ALTER that are not included in the
> defining query of the Materialized Table—who is responsible for updating
> the data in these new columns?
>
> when we detect some new columns which are not present in query,
> then
> 1) validate that the type of each of them is nullable (otherwise throw
> ValidationException)
> 2) merge them into schema
> 3) rewrite materializedTable query in a way that now this query fills
> newly added columns with nulls.
> It means that newly rewritten query will be responsible for filling
> these null values
> The approach is similar to the way CTAS behaves in the same situation.
>
> Probably one of the main use cases here is column name reservation for
> the future.
>
> >For ALTER MATERIALIZED TABLE operations like ADD/MODIFY/DROP
> columns—these changes could cause the pipeline bound to the Materialized
> Table to fail.
>
> Isn't it the same in the case of tables and pipelines bound to them?
> I was thinking that since there is MaterializedTableManager, then
> based on coming TableChangeOperation
> it could decide then how to process such change: for example full
> recompute or something more sophisticated
>
> Looking forward for your comments
>
> On Thu, Oct 9, 2025 at 11:13 AM Ron Liu <[email protected]> wrote:
> >
> > Hi, Sergey.
> >
> > I was on vacation recently, so sorry for joining this discussion so late.
> >
> > I’ve carefully reviewed the FLIP, and purely from the perspective of
> > aligning Materialized Table operations with those of a regular Table, I
> > support this proposal in principle. However, in my understanding,
> > Materialized Tables and regular Tables are fundamentally different. A
> > Materialized Table is bound to a specific pipeline that updates its
> > data—this pipeline is generated from the associated query. In contrast, a
> > regular Table isn’t tied to any pipeline; users manually write queries to
> > update its data. Performing an ALTER operation on a regular Table only
> > modifies metadata, whereas performing ALTER on a Materialized Table
> affects
> > not only metadata but also the underlying data update mechanism.
> >
> > Given this context, I have the following questions:
> > 1. When adding new columns via CREATE or ALTER that are not included in
> the
> > defining query of the Materialized Table—who is responsible for updating
> > the data in these new columns? I’m unclear about the purpose and use case
> > for adding such columns. Could you provide a concrete example?
> > 2. For ALTER MATERIALIZED TABLE operations like ADD/MODIFY/DROP
> > columns—these changes could cause the pipeline bound to the Materialized
> > Table to fail. What is the exact execution flow for these operations?
> Could
> > you elaborate on the runtime behavior for each type of operation? Since
> > these actions impact actual data updates—not just metadata—this is a
> > critical concern.
> > In summary, I believe we shouldn’t blindly apply all regular Table
> > operations directly to Materialized Tables. Instead, we should
> selectively
> > support a subset of operations based on real-world usage scenarios and
> > semantic correctness. What’s your take on this? Best, Ron
> >
> > Sergey Nuyanzin <[email protected]> 于2025年10月8日周三 08:05写道：
> >
> > > Hi Lincoln,
> > >
> > > Thank you for your feedback.
> > >
> > > I guess we already have similar behavior for CTAS, where we could put
> > > more columns than we have for query.
> > > In this case these extra columns should be filled with nulls, and the
> > > query should be rewritten accordingly [1].
> > > This also means that extra columns should have nullable type (there is
> > > a dedicated validation for this).
> > > It means that for non query columns we have such default values and
> > > query is rewritten taking them into account
> > >
> > > Regarding adding columns with alter, or some other changes like
> > > adding/dropping columns, constraints, distribution
> > > if I understand correctly MaterializedTableManager looking at table
> > > change can decide whether it should recompute materialized table or
> > > not
> > >
> > > Would it make sense?
> > >
> > > [1]
> > >
> https://github.com/apache/flink/blob/3478ddf08bce49e271f69b922a37ccada6f58688/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/table/SqlCreateTableAsConverter.java#L66-L74
> > >
> > >
> > > On Tue, Oct 7, 2025 at 4:14 AM Lincoln Lee <[email protected]>
> wrote:
> > > >
> > > > Thanks Sergey for driving this FLIP, it's a great addition to
> > > materialized
> > > > table!
> > > >
> > > > Since it coincided with China's National Day holiday and everyone is
> > > still
> > > > on
> > > > vacation, we couldn't reply promptly.
> > > >
> > > > I haven't fully reviewed all the content in the FLIP yet, but
> there's an
> > > > important issue on the ALTER statement:
> > > >
> > > > Unlike a regular CREATE TABLE, Materialized Table derives its schema
> from
> > > > the defined query, columns are generated based on the query (and,
> similar
> > > > to a
> > > > materialized view, the underlying data for these columns is tightly
> > > coupled
> > > > to
> > > > the query definition). Therefore, we cannot simply interpret the
> effect
> > > of
> > > > an
> > > > single `ALTER MATERIALIZED TABLE ADD New_Column` statement.
> Supporting
> > > this
> > > > likely requires accompanying column default value，and raises
> > > compatibility
> > > > concerns regarding historical data, that is a complex topic we
> previously
> > > > discussed offline during the design process of FLIP-492.
> > > >
> > > > Also, once Ron is back in the office, he may give a more detailed
> > > comment.
> > > >
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > Sergey Nuyanzin <[email protected]> 于2025年10月2日周四 20:15写道：
> > > >
> > > > > Thank you Ramin
> > > > >
> > > > > In case there is no more feedback/objections
> > > > > I would start voting thread next week
> > > > >
> > > > > On Thu, Sep 25, 2025 at 10:43 AM Ramin Gharib <
> [email protected]>
> > > > > wrote:
> > > > > >
> > > > > > Hi Sergey,
> > > > > > Thanks for driving this! This sounds good to me! +1
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Ramin
> > > > > >
> > > > > > On Wed, Sep 24, 2025 at 2:14 PM Sergey Nuyanzin <
> [email protected]
> > > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > > I'd like to start a discussion of FLIP-550
> > > > > > > Add similar support for CREATE/ALTER operations for
> MATERIALIZED
> > > > > > > TABLEs as for TABLEs [1].
> > > > > > >
> > > > > > > This FLIP is another step towards making tables and
> materialized
> > > > > > > tables more consistent. There was already one improvement in
> that
> > > > > > > direction like FLIP-542 [2] to add DISTRIBUTION and SHOW
> > > MATERIALIZED
> > > > > > > TABLES support. However there were several more things noticed
> > > > > > > comparing behavior for CREATE and ALTER operations. For
> instance
> > > right
> > > > > > > now for materialized tables it is impossible to set anything
> but
> > > table
> > > > > > > constraint while for tables (CREATE TABLE AS) it is possible to
> > > > > > > provide schema definition since FLIP-463 [3], also ALTER
> operations
> > > > > > > for TABLE is a way more mature than for MATERIALIZED TABLE.
> This
> > > FLIP
> > > > > > > is about to decrease the difference by enabling more similar
> > > features
> > > > > > > for materialized tables.
> > > > > > >
> > > > > > > Introducing schema definition support for materialized tables
> will
> > > > > > > provide users with greater control and flexibility and also
> will
> > > unify
> > > > > > > usage of tables and materialized tables.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=387648095
> > > > > > >
> > > > > > > [2]
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-542%3A+Make+materialized+table+DDL+consistent+with+regular+tables
> > > > > > >
> > > > > > > [3]
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-463%3A+Schema+Definition+in+CREATE+TABLE+AS+Statement
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Sergey
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Sergey
> > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Sergey
> > >
>
>
>
> --
> Best regards,
> Sergey
>
