Hey folks, Based on Yufei’s design doc and what we discussed during the sync, I shared my thoughts on what can be efficiently supported right now.
https://github.com/apache/iceberg/issues/3941#issuecomment-1061153554 <https://github.com/apache/iceberg/issues/3941#issuecomment-1061153554> I’d be interested to learn more about specific use cases that would violate the assumptions I listed in my comment. If you have such a use case in mind, please, comment on the issue. - Anton > On 24 Feb 2022, at 14:57, Yufei Gu <[email protected]> wrote: > > Hi everyone, > > Move the CDC design discussion to next week's Friday(Mar 4), 9-10am PST due > to an unexpected event. The meeting link will be the same, > meet.google.com/vam-cmfx-feo <http://meet.google.com/vam-cmfx-feo>. Thanks! > > Best, > > Yufei > > > On Tue, Feb 22, 2022 at 12:25 PM Yufei Gu <[email protected] > <mailto:[email protected]>> wrote: > Hi everyone, > > It's great to see a lot of interest in the design. > We are planning to have a meeting to discuss Iceberg CDC design on > Friday(2/25) 9-10am PST. The meeting link is meet.google.com/vam-cmfx-feo > <http://meet.google.com/vam-cmfx-feo>. We will talk about the general idea, > as well as open questions. The meeting will be recorded. > Best, > Yufei > > > On Fri, Feb 11, 2022 at 3:54 PM Holden Karau <[email protected] > <mailto:[email protected]>> wrote: > Oh cool, I have not had a chance to review much of this, but I was having a > conversation with a team which wanted similar features for a table so excited > to see folks working on it 👍 > > On Fri, Feb 11, 2022 at 12:40 PM Yufei Gu <[email protected] > <mailto:[email protected]>> wrote: > Hi team, > > We propose a way to generate the CDC records from the Iceberg tables. It is > an approach without table spec change and write-time logging. It will cover > the majority of CDC use cases, but no guarantee to all of them. We believe > it's a good start point to approach CDC in the Iceberg. Any feedback is > welcome! > https://docs.google.com/document/d/1bN6rdLNcYOHnT3xVBfB33BoiPO06aKBo56SZmuU9pnY/edit?usp=sharing > > <https://docs.google.com/document/d/1bN6rdLNcYOHnT3xVBfB33BoiPO06aKBo56SZmuU9pnY/edit?usp=sharing> > > Best, > > Yufei > -- > Twitter: https://twitter.com/holdenkarau <https://twitter.com/holdenkarau> > Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 > <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > <https://www.youtube.com/user/holdenkarau>
