Mika Naylor created FLINK-41468:
-----------------------------------
Summary: Converting table to materialized table with different
partition keys makes existing data unreadable
Key: FLINK-41468
URL: https://issues.apache.org/jira/browse/FLINK-41468
Project: Flink
Issue Type: Bug
Components: Table SQL / Gateway
Affects Versions: 2.4.0
Reporter: Mika Naylor
Fix For: 2.4.0
CREATE OR ALTER MATERIALIZED TABLE can turn a regular table into an MT, which
keeps the old table's data, but its partition keys come only from the statement
and nothing ensures that they match the old table's partition keys.
If theyre different, the existing data no longer match the MT's partitioning.
The conversion succeeds without a warning, but afterwards:
* If the partition key changes, batch reads of the MT fail.
* If an unpartitioned table is converted with PARTITION BY, batch reads fail.
* If a partitioned table is converted without PARTITIONED BY, reads work, but
rows written before the conversion lose their parition column value (it becomes
NULL).
FLIP-578 says that partition keys are inherited from the existing table, unless
the statement overrides them, but the current impl does not inherit them.
h4. Reproduction Steps
On 2.4 with table.materialized-table.conversion-from-table.enabled: true:
{code:java}
CREATE CATALOG mt_cat WITH ('type' = 'test-filesystem', 'path' =
'/tmp/flip578/catalog', 'default-database' = 'mydb');USE CATALOG mt_cat;CREATE
TABLE orders (user_id BIGINT NOT NULL, amount BIGINT, ds STRING) WITH ('format'
= 'json');INSERT INTO orders VALUES (1, 30, '2026-11-02');-- A regular table
partitioned by ds, with one row
CREATE TABLE p (ds STRING, user_id BIGINT NOT NULL, amount BIGINT)
PARTITIONED BY (ds) WITH ('format' = 'json');INSERT INTO p VALUES
('2026-10-31', 100, 500);-- Convert it, partitioned by a different key
CREATE OR ALTER MATERIALIZED TABLE p
PARTITIONED BY (user_id)
WITH ('format' = 'json')
FRESHNESS = INTERVAL '30' SECOND
AS SELECT ds, user_id, amount FROM orders;SET 'execution.runtime-mode' =
'batch';
SELECT * FROM p;
-- TableException: Partition keys are: [user_id], incomplete partition spec:
{ds=2026-10-31}
{code}
h4. Expected Behaviour
After a successful conversion, I would expect data the table held before
conversion could still be read, and if the conversion can't guarantee that, it
should be rejected with a clear error.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
