fightBoxing commented on PR #15282: URL: https://github.com/apache/iceberg/pull/15282#issuecomment-3957196182
> > Thanks for the PR @fightBoxing! > > > > 1. Do you mind briefly describing the approach taken in this PR? I'm assuming this does some kind of merge-on-read. What is the general architecture? Are there any limitations to the approach taken in this PR? > > 2. Could you remove all the files except for Flink 2.1? We usually merge support for the latest version first and then backport to the older ones. > > 1. CDC Functional Architecture Overview > Methodology Used: Changelog Inference Based on Snapshot Metadata (Not Merge-on-Read) > This PR is not a traditional merge-on-read solution. It uses a method of inferring changelogs based on Iceberg snapshot metadata. By analyzing the state changes (ADDED/DELETED) of manifest entries between two snapshots, it derives INSERT and DELETE change events and transforms them into a CDC stream with RowKind tags on the Flink side. > > <img alt="Clipboard_Screenshot_1771993182" width="923" height="1616" src="https://private-user-images.githubusercontent.com/26193074/554527427-aaf8d08a-be44-449a-b2ca-12227c4bdf95.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzIwMDIwNTcsIm5iZiI6MTc3MjAwMTc1NywicGF0aCI6Ii8yNjE5MzA3NC81NTQ1Mjc0MjctYWFmOGQwOGEtYmU0NC00NDlhLWIyY2EtMTIyMjdjNGJkZjk1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjI1VDA2NDIzN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEyOWNmY2I4ODllNzZlMTQwYjIxOTdiZTk0NWY4Y2I0OTI1YzFkN2M3NjExNGExZWRiMWE2YzcwMzEwMTgyM2QmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.f5y4rxEFDoAxCfL9NiEltqGco7YhVjafyqQQvLEVGlQ";> 2. The PR has been updated, but the Flink-2.1 version has been retained. [146193c](https://github.com/apache/iceberg/pull/15282/commits/146193c799708f10d8fba6dfb041ad3d3cb1405f) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://private-user-images.githubusercontent.com/26193074/554527427-aaf8d08a-be44-449a-b2ca-12227c4bdf95.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzIwMDIwNTcsIm5iZiI6MTc3MjAwMTc1NywicGF0aCI6Ii8yNjE5MzA3NC81NTQ1Mjc0MjctYWFmOGQwOGEtYmU0NC00NDlhLWIyY2EtMTIyMjdjNGJkZjk1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjI1VDA2NDIzN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEyOWNmY2I4ODllNzZlMTQwYjIxOTdiZTk0NWY4Y2I0OTI1YzFkN2M3NjExNGExZWRiMWE2YzcwMzEwMTgyM2QmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.f5y4rxEFDoAxCfL9NiEltqGco7YhVjafyqQQvLEVGlQ"](https://private-user-images.githubusercontent.com/26193074/554527427-aaf8d08a-be44-449a-b2ca-12227c4bdf95.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzIwMDIwNTcsIm5iZiI6MTc3MjAwMTc1NywicGF0aCI6Ii8yNjE5MzA3NC81NTQ1Mjc0MjctYWFmOGQwOGEtYmU0NC00NDlhLWIyY2EtMTIyMjdjNGJkZjk1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjI1VDA2NDIzN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEyOWNmY2I4ODllNzZlMTQwYjIxOTdiZTk0NWY4Y2I0OTI1YzFkN2M3NjExNGExZWRiMWE2YzcwMzEwMTgyM2QmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.f5y4rxEFDoAxCfL9NiEltqGco7YhVjafyqQQvLEVGlQ"){kind=link}