Sounds great, thank you for the details, Adam! As I'm a PMC member at Apache Flume, I think me helping out with a Debezium integration is a really good fit for me as I should be somewhat familiar with streaming log data. I've posted at dev@community ([1]). Let's see how it goes.
Donat [1]: https://lists.apache.org/thread.html/rc3a6bc4ca017a76c2e53b94d3faaa2f088963cbe56b608b2e6e4485c%40%3Cdev.community.apache.org%3E On Sat, Apr 3, 2021 at 12:32 AM Adam Kocoloski <a...@kocolosk.net> wrote: > > Sure thing. > > https://debezium.io is an open-source project largely developed by Red Hat > that standardizes change data capture across a variety of data sources. It > ships with connectors for Oracle, SQL Server, MySQL, PostgreSQL, and MongoDB, > and typically records changes into topics in Apache Kafka. > > CouchDB, of course, has a change capture feed as a public API endpoint, so a > connector should be a smaller effort than consuming the WAL from a > traditional RDMS. Integrating with Debezium would provide an easy way to > translate the _changes feed into a Kafka topic which plugs us into a much > larger ecosystem of tools and alleviates the need for every consumer of data > in CouchDB to build a bespoke “follower” of the _changes feed. > > Happy to dig up answers to other questions and to make connections with Kafka > / Debezium experts where it makes sense. Cheers, > > Adam > > > On Apr 2, 2021, at 5:44 PM, Bessenyei Balázs Donát <bes...@apache.org> > > wrote: > > > > Thank you for sharing the idea, Adam. I like it! > > Can you please share some details? > > > > > > Donat > > > > On Wed, Mar 31, 2021 at 2:14 AM Adam Kocoloski <kocol...@apache.org> wrote: > >> > >> I would bias towards ecosystem integrations, e.g. a metrics exporter for > >> Prometheus (I think one is in the works), or a Debezium connector for the > >> _changes feed (I haven’t seen any movement here). I suppose the Debezium > >> connector would ideally be part of the Debezium project itself, but that > >> feels like one that is reasonably self-contained and would provide a good > >> return on investment. Happy to help expand on that one a bit if you think > >> it’s worthwhile. > >> > >> Adam > >> > >>> On Mar 30, 2021, at 4:42 AM, Bessenyei Balázs Donát <bes...@apache.org> > >>> wrote: > >>> > >>> Wow, I didn't realize we are that far into the timeline. > >>> > >>> I actually don't have a well-scoped idea for GSoC this year. > >>> Does anyone have good candidates for a project? > >>> > >>> > >>> Donat > >>> > >>> On Mon, Mar 29, 2021 at 6:23 PM Joan Touzet <woh...@apache.org> wrote: > >>>> > >>>> https://community.apache.org/gsoc.html might be of help. > >>>> > >>>> If you are looking to talk to other Apache projects that have done this > >>>> before, you could reach out on the dev@community.a.o list. > >>>> > >>>> > >>>> https://lists.apache.org/thread.html/r189a563fe003ad8f0e4c298e18fad4da8d4b2854bd2a5d741ae3ac45%40%3Cdev.community.apache.org%3E > >>>> > >>>> Note the dependency on JIRA: > >>>> > >>>>> All ASF projects are invited to submit their ideas to their issue > >>>>> tracker, please be sure to add the labels “gsoc2021” and “mentor” so > >>>>> that we can automatically include them in our list of subjects. If your > >>>>> project does not use JIRA please contact d...@community.apache.org. > >>>> > >>>> so you'll need to post at dev@community to get included in the master > >>>> list. > >>>> > >>>> Student applications start tomorrow for 2 weeks, so you'll need to get a > >>>> move on... If I'm around on chat I can try and help a bit. > >>>> > >>>> -Joan > >>>> > >>>> On 29/03/2021 01:20, Bessenyei Balázs Donát wrote: > >>>>> If there are any projects that don't exceed my CouchDB / erlang / JS > >>>>> knowledge, I'd make sure I'm available enough to support someone doing > >>>>> a GSoC with us. > >>>>> What's the workflow here? Do we have to apply as a project? Do we have > >>>>> to propose projects? > >>>>> I did look at "Prospective ASF mentors: read this" of [1], but I don't > >>>>> see what it looks like for a project. Do we need a vote here? > >>>>> > >>>>> > >>>>> Donat > >>>>> > >>>>> [1]: https://community.apache.org/gsoc.html > >>>>> > >>>>> On Sun, Mar 28, 2021 at 11:52 PM Joan Touzet <woh...@apache.org> wrote: > >>>>>> > >>>>>> The ASF often ends up doing GSoC. I don't think we've ever had the > >>>>>> sponsor within the project for it (or for Outreachy, for that matter). > >>>>>> > >>>>>> The most critical part is being available on a regular basis for proper > >>>>>> mentoring. If you don't think you can get that into your schedule, > >>>>>> don't > >>>>>> volunteer. Assume you will get zero support from any other developer > >>>>>> (not true, but best to plan for the worst case situation...) > >>>>>> > >>>>>> The second most critical part is to come up with a self-contained > >>>>>> project that makes sense for CouchDB. The most obvious thing to me > >>>>>> would > >>>>>> be Fauxton work, esp. as it falls into the "sweet spot" of JS > >>>>>> development. I dunno how good of a target main is, given how in flux it > >>>>>> is; others might have a better take on that. There's also this PR that > >>>>>> never finished up: > >>>>>> > >>>>>> https://github.com/apache/couchdb/issues/1254 > >>>>>> > >>>>>> These topics are all probably too big, but maybe one of them could be > >>>>>> cut down to something summer-sized: > >>>>>> > >>>>>> https://github.com/apache/couchdb/projects/1 > >>>>>> > >>>>>> Thanks for taking on this initiative! I know for a fact I won't have > >>>>>> time this summer, or I'd agree to join you. > >>>>>> > >>>>>> -Joan > >>>>>> > >>>>>> On 28/03/2021 15:59, Bessenyei Balázs Donát wrote: > >>>>>>> Hi All, > >>>>>>> > >>>>>>> I've just seen that the ASF is accepted as a mentoring organisation > >>>>>>> for GSoC 2021. > >>>>>>> Is CouchDB interested in participating? > >>>>>>> I've never done a GSoC before, but I'd certainly be interested. I'd be > >>>>>>> happy to help a student contribute to CouchDB. > >>>>>>> > >>>>>>> What do you all think? > >>>>>>> > >>>>>>> > >>>>>>> Thank you, > >>>>>>> Donat > >>>>>>> > >> >
