Thorne created FLINK-40323:
------------------------------
Summary: [cdc-common] Support JSON Data Type in Flink CDC
Key: FLINK-40323
URL: https://issues.apache.org/jira/browse/FLINK-40323
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Reporter: Thorne
*1、Background & Problem Description*
In real-time data synchronization scenarios, most business databases widely use
*JSON/JSONB data types* to store flexible, unstructured and semi-structured
business data, such as user extension attributes, order additional information,
device metadata and dynamic business parameters.
Currently, Flink CDC connectors lack native and standardized support for
database JSON data types. For table fields defined as JSON/JSONB in source
databases, the existing processing logic has the following problems: *
JSON fields are forcibly parsed as plain string types, losing original JSON
structural attribute information
*
Inconsistent JSON parsing rules across different database connectors, resulting
in data synchronization deviation
*
Cannot dock with Flink-CDC native JSON data type, failing to support subsequent
JSON function calculations and structured analysis
The above limitations greatly restrict the capability of Flink CDC in
semi-structured data real-time synchronization and analysis, and cannot meet
the flexible business data synchronization demands of modern data warehouses
and real-time computing platforms.
h3. 2、 Requirement
We need to add *native JSON data type support* for Flink CDC core connectors,
to implement full-process standardized parsing, identification and
synchronization of database JSON/JSONB fields, and perfectly dock with Flink
built-in JSON data types and JSON computing functions.
h3. 3、Core Capabilities
*
{*}Native Type Identification{*}: Automatically identify JSON/JSONB type fields
in source database tables during schema parsing, distinguish them from ordinary
string fields
*
{*}Standardized Data Parsing{*}: Parse original database JSON binary/string
data into Flink native JSON type objects, retain complete JSON structure
(key-value pairs, nested structures, arrays)
*
{*}Cross-database Compatibility{*}: Uniform parsing rules for JSON types of
mainstream databases (MySQL JSON, PostgreSQL JSONB, etc.) to eliminate
cross-database parsing differences
*
{*}Complete Change Capture{*}: Support capturing full JSON field changes (full
update of JSON content, partial update of nested fields) in CDC incremental
synchronization
*
{*}Compatible with Existing Logic{*}: Provide compatible configuration items,
support fallback to string parsing mode, ensure no impact on existing online job
h2. 4. Related Tasks
*
Optimize CDC schema parsing logic to add JSON type mapping rules
*
Rewrite JSON field deserialization logic for each connector
*
Add compatible configuration parameters and documentation updates
*
Supplement end-to-end test cases for JSON data type synchronizatio
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
