Hi! I'm glad to hear updates about this project from your end! On our end I did not manage to get great progress as I dealt with infra and the python package etc. However I have been tinkering with the idea of writing a Flink job that does diff scans and writes those results to a DR Kudu cluster. This is still in POC state. I'm happy to hear that you've managed to use your proposal in production that is great news! Internally we've talked about this and one key point is that it would be probably best if we implemented a proper CDC interface for Kudu. This could be then used to satisfy your requirement as well, right? (eg picking changes up in Kafka - please correct me if I'm wrong here) Personally I think if you have promising results from prod deployments its a great thing, and it benefits the project and the project health. It's been quite a while and I would like to take a couple of days and understand(again) your proposal, for this I have a couple of questions: - are there any updates to the design? (is this up to date: https://docs.google.com/document/d/1ihqPFO1vulpYDYYKcHmhes0LCOXKQtdAf5ub0y2LiaM/edit ?) - is https://gerrit.cloudera.org/#/c/19909/ that went to prod, or did you apply any other changes eg improvements, fixes? - have you experienced any pitfalls with the design/implementation?
Anyway will go through the design doc again, and post questions, and we can continue the discussion in this thread. Thank you! Marton 邓科 <kdeng...@gmail.com> ezt írta (időpont: 2024. okt. 8., K, 9:21): > Hello, Marton. > > It's been a while since you sent this email, and I'm wondering if there has > been any progress. > Based on the commit we made at https://gerrit.cloudera.org/#/c/19909/, > we've implemented a relatively stable version that has been running stably > in dozens of customer environments for a long time. We are now considering > whether to contribute this feature back to the community. If you could > share your progress and future plans, we can determine whether there is > still a need to submit this part of the code. > > Thanks, > Ke Deng > > Marton Greber <greber...@gmail.com> 于2023年11月14日周二 23:10写道： > > > Devs, > > > > > > We have been tinkering with a proof of concept(POC) to accomplish cross > > cluster async replication. The use case is to have a backup Kudu cluster > > for disaster recovery. The new replication feature could be treated as an > > alternative to backup/restore, but with finer time granularity. Moreover > it > > would eliminate the need for intermediate storage. > > > > > > For our POC we have been looking for inspiration at YugaByte xCluster > > replication > > < > > > https://docs.yugabyte.com/preview/architecture/docdb-replication/async-replication/ > > > > > (the active-passive part). This would be a CDC based approach, where we > > have Kudu CDC producers/consumers. > > > > > > On the other hand while looking at https://gerrit.cloudera.org/c/19909/ > > "Support write ops to kafka with kafka client” I’ve found some > > similarities. Here, according to my understanding, the goal is to > > move records from Kudu into Kafka. > > > > > > I think there is an intersection between these two projects, and wanted > to > > start the conversation about potential ways to consolidate these two > > projects. Figuring out what are the commonalities, and thereby avoiding > > pushing in pieces of changes which are quite similar (bloating the > > codebase). > > > > > > — > > > > > > Some initial thoughts: > > > > - the need for a CDC interface for Kudu emerges as a commonality > > - https://gerrit.cloudera.org/c/19909/ could avoid adding the Kafka > > client into Kudu, by leveraging the above CDC interface in a Kafka > > source > > connector for example > > - this would maybe be a better separation of concerns > > - We could re-think our CDC approach such that it is a generic > interface > > rather than the Kudu:Kudu specific one. > > - Maybe, for our purpose, we could maybe initially reuse the Kudu -> > > Kafka (with connector) way, and implement async replication by > > implementing > > the other end: the Kafka sink connector > > > > — > > > > > > Let me know your thoughts on this one! > > > > > > Thanks, > > Marton > > >
