[
https://issues.apache.org/jira/browse/FLINK-41008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41008:
-----------------------------------
Labels: pull-request-available (was: )
> Support multi-table transaction stream load in StarRocks pipeline connector
> ---------------------------------------------------------------------------
>
> Key: FLINK-41008
> URL: https://issues.apache.org/jira/browse/FLINK-41008
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Reporter: peiyu
> Priority: Major
> Labels: pull-request-available
>
> h2. Background
> The StarRocks Flink Connector introduced multi-table transaction stream
> load in 1.2.15 (upstream PR
> StarRocks/starrocks-connector-for-apache-flink#487),
> with several hardening fixes in 1.2.16. It allows multiple tables written
> in the same flush cycle to share one StarRocks transaction label, so they
> become visible atomically, and reduces the number of transactions/labels
> from one-per-table to one-per-sink-subtask.
> Currently the Flink CDC StarRocks pipeline connector (after FLINK-41006,
> connector 1.2.16) does not expose this capability:
> * Each table still commits with its own independent transaction label.
> * Tables written in the same checkpoint/flush cycle are not visible
> atomically, which shows up as an inconsistency window on failure
> recovery (converged later by primary-key idempotent replays).
> * Wide-table-count sync jobs create many small transactions per flush
> cycle, adding FE pressure.
> h2. Proposal
> Expose the capability behind an opt-in option:
> * Add {{sink.transaction.multi-table.enabled}} (default {{false}}) and
> {{sink.transaction.multi-table.buffer-size}} options, passed through
> to the StarRocks connector. With the option disabled, behavior stays
> unchanged.
> * Fail fast at pipeline submission when the option is enabled but the
> StarRocks server version is below 4.0 or unreachable
> ({{SELECT CURRENT_VERSION()}} via JDBC).
> * Fail fast when schema change events target more than one database,
> since a shared StarRocks transaction is single-database scoped.
> * When enabled, mark every serialized row with a fixed source partition
> and a transaction-end flag, so all tables written by the same sink
> subtask within one flush cycle share one transaction label while
> keeping the current at-least-once flush cadence.
> h2. Limitations (to be documented)
> * Requires StarRocks server >= 4.0.
> * All target tables must be routed to the same StarRocks database.
> * Semantics stays at-least-once; duplicates on failure are deduplicated
> by the primary-key table.
> * Transaction scope = sink subtask: cross-subtask visibility is not
> synchronized (same granularity as the upstream connector).
> h2. Depends on
> FLINK-41006 (Upgrade StarRocks Flink connector to 1.2.16)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
