Hi Lu, Just to iterate the status quo: Flink supports upserts, but only via equality delete + append. So technically, "streaming writes" aren't an issue. It's the read path which causes the issue, because unlike positional deletes, which can be resolved on the fly during streaming reads, equality deletes potentially require a full table scan to be materialized. Constant snapshot compaction is required to keep the read path efficient.
>1. A Flink job that continuously appends CDC events into an append-only raw >table >2. A periodically scheduled Spark job that performs upsert the `current` table >using `raw` table This makes sense. Conceptually, you are pre-compacting upserts before writing into the final "current" table. This avoids equality deletes entirely and keeps the read path on the "current" table efficient at all times. The drawback is that your lower bound latency will be the interval at which the Spark job runs, but this is an acceptable price to pay, until we have a way to write positional deletes right away, avoiding equality deletes entirely. Cheers, Max On Wed, Jan 21, 2026 at 8:48 AM melin li <[email protected]> wrote: > > Flink CDC support reading binlog data from databases such as MySQL and > PostgreSQL, and writing it to Iceberg, Hudi, and Paimon. > https://github.com/apache/flink-cdc/pulls?q=iceberg > > Steven Wu <[email protected]> 于2026年1月21日周三 15:27写道： >> >> Lu, >> >> you are correct about the design doc for Flink writing position deletes >> only. The original design has high complexity. We were thinking about >> alternatives with narrower scope. But there isn't any progress and timeline . >> >> IMHO, your setup is a good practice today. Ryan wrote a series of blogs for >> the pattern: https://tabular.medium.com/hello-world-of-cdc-e6f06ddbfcc0. >> >> Some people use the current Flink Iceberg sink for CDC ingestion. But it >> would produce equality deletes that would require aggressive compactions and >> add operational burden too. Also not all engines can read equality deletes. >> >> Thanks, >> Steven >> >> On Tue, Jan 20, 2026 at 8:44 PM Gang Wu <[email protected]> wrote: >>> >>> Hi Lu, >>> >>> Nice to hear from you here in the Iceberg community :) >>> >>> We have built an internal service to stream upserts into position deletes >>> which happens to have a lot in common with [1] and [2]. I believe this is a >>> viable approach to achieve second freshness. >>> >>> [1] >>> https://docs.google.com/document/d/1Jz4Fjt-6jRmwqbgHX_u0ohuyTB9ytDzfslS7lYraIjk >>> [2] https://www.mooncake.dev/whitepaper >>> >>> Best, >>> Gang >>> >>> >>> >>> >>> On Wed, Jan 21, 2026 at 11:05 AM Lu Niu <[email protected]> wrote: >>>> >>>> Hi Iceberg community, >>>> >>>> What are the current best practices for streaming upserts into an Iceberg >>>> table? >>>> >>>> Today, we have the following setup in production to support CDC: >>>> >>>> 1. A Flink job that continuously appends CDC events into an append-only >>>> raw table >>>> 2, A periodically scheduled Spark job that performs upsert the `current` >>>> table using `raw` table >>>> >>>> We are exploring whether it’s feasible to stream upserts directly into an >>>> Iceberg table from Flink. This could simplify our architecture and >>>> potentially further reduce our data SLA. We’ve experimented with this >>>> approach before, but ran into reader-side performance issues due to the >>>> accumulation of equality deletes over time. >>>> >>>> From what I can gather, streaming upserts still seems to be an open design >>>> area: >>>> >>>> 1. (Please correct me if I’m wrong—this summary is partly based on ChatGPT >>>> 5.1.) The book “Apache Iceberg: The Definitive Guide” suggests the >>>> two-table pattern we’re currently using in production. >>>> 2. These threads: >>>> https://lists.apache.org/thread/gjjr30txq318qp6pff3x5fx1jmdnr6fv , >>>> https://lists.apache.org/thread/xdkzllzt4p3tvcd3ft4t7jsvyvztr41j discuss >>>> the idea of outputting only positional deletes (no equality deletes) by >>>> introducing an index. However, this appears to still be under discussion >>>> and may be targeted for v4, with no concrete timeline yet. >>>> 3. this thread >>>> https://lists.apache.org/thread/6fhpjszsfxd8p0vfzc3k5vw7zmcyv2mq talks >>>> about deprecating equality deletes, but I haven’t seen a clearly defined >>>> alternative come out of that discussion yet. >>>> >>>> Given all of the above, I’d really appreciate guidance from the community >>>> on: >>>> >>>> 1. Recommended patterns for streaming upserts with Flink into Iceberg >>>> today (it's good to know the long term possible as well, but my focus is >>>> what's possible in near term). >>>> 2. Practical experiences or lessons learned from teams running streaming >>>> upserts in production >>>> >>>> Thanks in advance for any insights and corrections. >>>> >>>> Best >>>> Lu
