Thank you for all your replies!
> Some people use the current Flink Iceberg sink for CDC ingestion. But it
would produce equality deletes that would require aggressive compactions
and add operational burden too
Since the main concern is reader-side performance degradation due to the
accumulation of equality deletes over time. Is there a way to estimate
impact on the reader side based on equality deletes in a snapshot summary?
```
while new_snapshot_summary_is_ready:
should_compact = analyze_snapshot_summary(snapshot_summary)
if should_compact:
rewrite_data_files()
```
> The original design has high complexity. We were thinking about
alternatives with narrower scope. But there isn't any progress and timeline
.
If this is the community aligned long term, Is there any way I could
contribute to speed this up? Thanks!
Best
Lu
On Wed, Jan 21, 2026 at 2:18 AM Maximilian Michels <[email protected]> wrote:
> Hi Lu,
>
> Just to iterate the status quo: Flink supports upserts, but only via
> equality delete + append. So technically, "streaming writes" aren't an
> issue. It's the read path which causes the issue, because unlike
> positional deletes, which can be resolved on the fly during streaming
> reads, equality deletes potentially require a full table scan to be
> materialized. Constant snapshot compaction is required to keep the
> read path efficient.
>
> >1. A Flink job that continuously appends CDC events into an append-only
> raw table
> >2. A periodically scheduled Spark job that performs upsert the `current`
> table using `raw` table
>
> This makes sense. Conceptually, you are pre-compacting upserts before
> writing into the final "current" table. This avoids equality deletes
> entirely and keeps the read path on the "current" table efficient at
> all times. The drawback is that your lower bound latency will be the
> interval at which the Spark job runs, but this is an acceptable price
> to pay, until we have a way to write positional deletes right away,
> avoiding equality deletes entirely.
>
> Cheers,
> Max
>
> On Wed, Jan 21, 2026 at 8:48 AM melin li <[email protected]> wrote:
> >
> > Flink CDC support reading binlog data from databases such as MySQL and
> PostgreSQL, and writing it to Iceberg, Hudi, and Paimon.
> > https://github.com/apache/flink-cdc/pulls?q=iceberg
> >
> > Steven Wu <[email protected]> 于2026年1月21日周三 15:27写道：
> >>
> >> Lu,
> >>
> >> you are correct about the design doc for Flink writing position deletes
> only. The original design has high complexity. We were thinking about
> alternatives with narrower scope. But there isn't any progress and timeline
> .
> >>
> >> IMHO, your setup is a good practice today. Ryan wrote a series of blogs
> for the pattern:
> https://tabular.medium.com/hello-world-of-cdc-e6f06ddbfcc0.
> >>
> >> Some people use the current Flink Iceberg sink for CDC ingestion. But
> it would produce equality deletes that would require aggressive compactions
> and add operational burden too. Also not all engines can read equality
> deletes.
> >>
> >> Thanks,
> >> Steven
> >>
> >> On Tue, Jan 20, 2026 at 8:44 PM Gang Wu <[email protected]> wrote:
> >>>
> >>> Hi Lu,
> >>>
> >>> Nice to hear from you here in the Iceberg community :)
> >>>
> >>> We have built an internal service to stream upserts into position
> deletes which happens to have a lot in common with [1] and [2]. I believe
> this is a viable approach to achieve second freshness.
> >>>
> >>> [1]
> https://docs.google.com/document/d/1Jz4Fjt-6jRmwqbgHX_u0ohuyTB9ytDzfslS7lYraIjk
> >>> [2] https://www.mooncake.dev/whitepaper
> >>>
> >>> Best,
> >>> Gang
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Jan 21, 2026 at 11:05 AM Lu Niu <[email protected]> wrote:
> >>>>
> >>>> Hi Iceberg community,
> >>>>
> >>>> What are the current best practices for streaming upserts into an
> Iceberg table?
> >>>>
> >>>> Today, we have the following setup in production to support CDC:
> >>>>
> >>>> 1. A Flink job that continuously appends CDC events into an
> append-only raw table
> >>>> 2, A periodically scheduled Spark job that performs upsert the
> `current` table using `raw` table
> >>>>
> >>>> We are exploring whether it’s feasible to stream upserts directly
> into an Iceberg table from Flink. This could simplify our architecture and
> potentially further reduce our data SLA. We’ve experimented with this
> approach before, but ran into reader-side performance issues due to the
> accumulation of equality deletes over time.
> >>>>
> >>>> From what I can gather, streaming upserts still seems to be an open
> design area:
> >>>>
> >>>> 1. (Please correct me if I’m wrong—this summary is partly based on
> ChatGPT 5.1.) The book “Apache Iceberg: The Definitive Guide” suggests the
> two-table pattern we’re currently using in production.
> >>>> 2. These threads:
> https://lists.apache.org/thread/gjjr30txq318qp6pff3x5fx1jmdnr6fv ,
> https://lists.apache.org/thread/xdkzllzt4p3tvcd3ft4t7jsvyvztr41j discuss
> the idea of outputting only positional deletes (no equality deletes) by
> introducing an index. However, this appears to still be under discussion
> and may be targeted for v4, with no concrete timeline yet.
> >>>> 3. this thread
> https://lists.apache.org/thread/6fhpjszsfxd8p0vfzc3k5vw7zmcyv2mq talks
> about deprecating equality deletes, but I haven’t seen a clearly defined
> alternative come out of that discussion yet.
> >>>>
> >>>> Given all of the above, I’d really appreciate guidance from the
> community on:
> >>>>
> >>>> 1. Recommended patterns for streaming upserts with Flink into Iceberg
> today (it's good to know the long term possible as well, but my focus is
> what's possible in near term).
> >>>> 2. Practical experiences or lessons learned from teams running
> streaming upserts in production
> >>>>
> >>>> Thanks in advance for any insights and corrections.
> >>>>
> >>>> Best
> >>>> Lu
>
