I can give an update on using branches vs using tables. I originally wanted to use branches, but I now prefer the approach using separate tables.
The main reason to use a separate table is to avoid schema differences between branches. We recently looked into whether to support different schemas across table branches and decided not to pursue it. The problem is how to reconcile schema changes between branches. If you run ADD COLUMN on two different branches with the same column name, should Iceberg use the same field ID for both? There are cases where it should (two audited commits) and cases where it should not (testing code revisions). I don't think we can tell when adding the column which decision is correct, and getting it wrong is a correctness problem. Since it is a problem to have different schemas across branches, it's also a problem to keep CDC metadata and older records in a branch of the merged mirror table. If you wanted to do this, you can use an optional struct of CDC metadata that you can set to null in the main branch, but you'd still have to decide how to handle dropped columns. I think most people don't want to have dropped columns in the mirror table, but if you're looking at the changelog table you do want them there since they're part of the historical record. It would be nice to have a single table, but that doesn't seem worth the extra headache of handling extra schema issues and committing more often (increasing the risk of conflicts). The multi-table solution also fits better with materialized views, eventually. Ryan On Sun, Feb 25, 2024 at 5:43 PM Manu Zhang <owenzhang1...@gmail.com> wrote: > Thanks Ryan for the updates! I've already read your blog series and > learned a lot. > Our customer is using a similar Flink Append + Spark MergeInto approach. > I'm wondering whether there's a plan to implement this pattern with two > branches as proposed by Jack in Improvement 3. > > Manu > > > > On Mon, Feb 26, 2024 at 5:13 AM Ryan Blue <b...@tabular.io> wrote: > >> Manu, >> >> I haven't seen much improvement to the Flink/UPSERT approach to CDC. It's >> still half-finished. There are some efforts to add table maintenance to >> Flink, but the main issues -- work is being deferred and never done -- >> haven't been addressed. I don't recommend this approach. >> >> The other approach has seen progress. I wrote a blog series >> <https://tabular.io/blog/cdc-zen-art-of-cdc-performance/> about CDC >> patterns that builds to the pattern of using separate changelog and mirror >> tables to help people understand how to do it and what the benefits are. We >> are also getting closer to a release with views that will allow us to use >> the latest from the changelog table at read time. That's going to be the >> most efficient implementation overall. >> >> Ryan >> >> On Tue, Feb 20, 2024 at 1:41 AM Manu Zhang <owenzhang1...@gmail.com> >> wrote: >> >>> Bump up this thread again. Are we actively working on any proposed >>> approaches? >>> >>> Manu >>> >>> On Fri, May 5, 2023 at 9:14 AM Ryan Blue <b...@tabular.io> wrote: >>> >>>> Thanks for taking the time to write this up, Jack! It definitely >>>> overlaps my own thinking, which is a good confirmation that we're on the >>>> right track. There are a couple of things I want to add to the discussion. >>>> >>>> First, I think the doc relies fairly heavily on the Iceberg/Flink >>>> approach being _the_ approach to CDC with Iceberg. That's not how I've seen >>>> the pattern implemented at scale, and I generally think of the >>>> Iceberg/Flink implementation as unfinished -- it is missing critical >>>> components. >>>> >>>> As a format, Iceberg tries to be flexible and allow you to make >>>> trade-offs. The main trade-off you get is to defer work until later. >>>> Sorting is a good example, where Flink can't sort easily so it leaves >>>> sorting for downstream systems. Another way we defer work is using delete >>>> files to keep write amplification down. And yet another way is using >>>> equality delete files to avoid needing to locate records. The issue with >>>> the Iceberg/Flink approach is that it uses all 3 of these and defers a ton >>>> of work that never gets completed. I think the Iceberg/Flink UPSERT >>>> feature is incomplete and would not recommend it without something cleaning >>>> up the tables. >>>> >>>> It seems to me that Approaches 1 and 2 are trying to fix this, but not >>>> really in straightforward ways. I like that Approach 1 also preserves >>>> history, but I think there's a better and more direction option in Approach >>>> 3. >>>> >>>> Second, I think we also need to consider transactional consistency. >>>> This isn't hard to achieve and is how people consuming the table think >>>> about the data. I think we should always try to mirror consistency in the >>>> source table downstream in the mirror table. >>>> >>>> In the end, I think we probably agree on the overall approach (3). Keep >>>> a changelog branch or table and a mirror table, then keep them in sync. I'd >>>> also add views to the mix to get the latest up-to-date information. We can >>>> also make this better by adapting Approach 2 for streaming writes. I think >>>> Anton has been working on an index approach that would work. >>>> >>>> If we're aligned, I think it should be easy to start building this >>>> pattern and adding support in Iceberg for things like updating the schemas >>>> at the same time. >>>> >>>> Ryan >>>> >>>> On Thu, May 4, 2023 at 3:00 PM Steven Wu <stevenz...@gmail.com> wrote: >>>> >>>>> Thanks Jack for the great write-up. Good summary of the current >>>>> landscape of CDC too. Left a few comments to discuss. >>>>> >>>>> On Wed, Apr 26, 2023 at 11:38 AM Anton Okolnychyi >>>>> <aokolnyc...@apple.com.invalid> wrote: >>>>> >>>>>> Thanks for starting a thread, Jack! I am yet to go through the >>>>>> proposal. >>>>>> >>>>>> I recently came across a similar idea in BigQuery, which relies on a >>>>>> staleness threshold: >>>>>> >>>>>> https://cloud.google.com/blog/products/data-analytics/bigquery-gains-change-data-capture-functionality/ >>>>>> >>>>>> It would also be nice to check if there are any applicable ideas in >>>>>> Paimon: >>>>>> https://github.com/apache/incubator-paimon/ >>>>>> >>>>>> - Anton >>>>>> >>>>>> On Apr 26, 2023, at 11:32 AM, Jack Ye <yezhao...@gmail.com> wrote: >>>>>> >>>>>> Hi everyone, >>>>>> >>>>>> As we discussed in the community sync, it looks like we have some >>>>>> general interest in improving the CDC streaming process. Dan mentioned >>>>>> that >>>>>> Ryan has a proposal about an alternative CDC approach that has an >>>>>> accumulated changelog that is periodically synced to a target table. >>>>>> >>>>>> I have a very similar design doc I have been working on for quite >>>>>> some time to describe a set of improvements we could do to the Iceberg >>>>>> CDC >>>>>> use case, and it contains a very similar improvement (see improvement 3). >>>>>> >>>>>> I would appreciate feedback from the community about this doc, and I >>>>>> can organize some meetings to discuss our thoughts about this topic >>>>>> afterwards. >>>>>> >>>>>> Doc link: >>>>>> https://docs.google.com/document/d/1kyyJp4masbd1FrIKUHF1ED_z1hTARL8bNoKCgb7fhSQ/edit# >>>>>> >>>>>> Best, >>>>>> Jack Ye >>>>>> >>>>>> >>>>>> >>>> >>>> -- >>>> Ryan Blue >>>> Tabular >>>> >>> >> >> -- >> Ryan Blue >> Tabular >> > -- Ryan Blue Tabular
