tchivs created FLINK-38221:
------------------------------
Summary: PostgreSQL CDC fails to capture data from partition
tables when configured with parent table names
Key: FLINK-38221
URL: https://issues.apache.org/jira/browse/FLINK-38221
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.4.0
Environment: PostgreSQL CDC with partitioned tables
Reporter: tchivs
Fix For: cdc-3.5.0
h2. Problem Description
h3. Critical Data Loss Bug
PostgreSQL CDC connector completely fails to capture any changelog data when
users configure partition tables using standard parent table names. This is a
*critical data loss bug* affecting production deployments.
h3. Reproduction Steps
# *Create a partitioned table in PostgreSQL:*
{{CREATE TABLE aia_t_icc_jjdb (
id BIGINT,
name VARCHAR(255),
partition_date DATE NOT NULL,
PRIMARY KEY (id, partition_date)
) PARTITION BY RANGE (partition_date);
CREATE TABLE aia_t_icc_jjdb_202401 PARTITION OF aia_t_icc_jjdb
FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');}}
# *Configure Flink CDC with parent table name:*
{{source:
type: postgres-cdc
hostname: localhost
port: 5432
username: postgres
password: password
database-name: testdb
schema-name: public
table-name: aia_t_icc_jjdb # Parent partition table}}
# *Insert data into the partition table:*
{{INSERT INTO aia_t_icc_jjdb (id, name, partition_date)
VALUES (1, 'test', '2024-01-15');}}
h3. Expected Behavior
* CDC should discover the parent partition table {{aia_t_icc_jjdb}}
* CDC should capture INSERT/UPDATE/DELETE operations on the partition table
* Changelog events should be emitted for data changes
h3. Actual Behavior
* *NO CDC data is captured* - complete data loss
* Parent partition table {{aia_t_icc_jjdb}} is filtered out during table
discovery
* CDC connector runs without errors but produces zero changelog events
* Users receive no indication that the table configuration is ineffective
h2. Root Cause Analysis
h3. Technical Root Cause
The issue is in {{TableDiscoveryUtils.java}} line 55-56:
{{Set<TableId> allTableIds = jdbc.readTableNames(
database, null, null, new String[] \{"TABLE"}); // Missing "PARTITIONED
TABLE"}}
h3. PostgreSQL Partition Table Structure
* {*}Parent partition tables{*}: {{pg_class.relkind = 'p'}} (partitioned table)
* {*}Child partition tables{*}: {{pg_class.relkind = 'r'}} (regular table)
* {*}Table discovery query{*}: Only looks for {{tabletype = 'TABLE'}}
* {*}Problem{*}: Parent partitions need {{tabletype = 'PARTITIONED TABLE'}} to
be discovered
h3. Discovery Failure Chain
# User configures {{table-name: "aia_t_icc_jjdb"}} (parent partition)
# TableDiscoveryUtils queries {{pg_tables WHERE tabletype IN ('TABLE')}}
# Parent partition table not found (it's type 'PARTITIONED TABLE')
# Table filter excludes the configured table name
# Result: Zero CDC data captured
h2. Current User Workarounds (and their problems)
h3. Workaround 1: Regex Pattern Matching
{{table-name: "aia_t_icc_jjdb_\\d\{6}" # Match child partitions}}
*Problems:*
* Extremely slow initialization (loads schema for every child partition)
* Complex configuration requiring internal partition naming knowledge
* Error-prone and maintenance-intensive
* Breaks when partition naming changes
h3. Workaround 2: List All Child Partitions
{{table-name: "aia_t_icc_jjdb_202401,aia_t_icc_jjdb_202402,..."}}
*Problems:*
* Manual maintenance required for new partitions
* Configuration becomes unmanageable with many partitions
* Still slower than parent table approach
h2. Impact Assessment
h3. Severity: Critical
* {*}Data Loss{*}: Complete CDC data loss for partition tables
* {*}Silent Failure{*}: No obvious error messages, difficult to diagnose
* {*}Production Impact{*}: Affects real-time data pipelines using partition
tables
* {*}User Experience{*}: Forces complex, error-prone configurations
h3. Affected Scenarios
* Any PostgreSQL CDC deployment using partitioned tables
* Range partitioned tables (most common use case)
* List partitioned tables
* Hash partitioned tables
* Multi-level partition hierarchies
h3. Business Impact
* Real-time analytics miss partition table data changes
* Data synchronization failures in production
* Increased operational complexity due to workarounds
* Potential compliance issues due to incomplete data capture
h2. Proposed Solution
h3. Core Fix
Enhance {{TableDiscoveryUtils.java}} to include partitioned tables:
{{Set<TableId> allTableIds = jdbc.readTableNames(
database, null, null, new String[] \{"TABLE", "PARTITIONED TABLE"});}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
