And thanks also to Kaustubh and Zoi for their input, indeed a generic JDBC sink would be something I can definitely look into while I am at it 😉
________________________________ From: Gavriilidis, Haralampos Sent: Tuesday, October 21, 2025 10:21 PM To: [email protected] Subject: Re: PostgreSQL Sink Hi Mirko, yes, the embedded PostgreSQL idea is only for testing purposes, and not related to the Sink operator. Most of the other platforms can be used as libraries (e.g., Spark), which makes testing easy. However, for PostgreSQL the tests require a separate instance running. Actually, there seem to be some PostgreSQL-related tests, but they are not run during the build, are they? In this case embedding would mean that we use one of the mentioned projects, to include PostgreSQL as a Maven dependency, only for the tests. Then we could also run PostgreSQL tests during the build etc. Not sure if these embedded PostgreSQL projects work well, but would be willing to try it out. Or is there an other standard way of testing the PostgreSQL platform within Wayang. The PostgreSQL Sink operator is a separate thing, that I want to bring from an older project. Best, Harry ________________________________ From: Mirko Kämpf <[email protected]> Sent: Tuesday, October 21, 2025 9:00:51 PM To: [email protected] Subject: Re: PostgreSQL Sink Hello Wayang fans ;-) Hello Harry, this embedded Postgres approach sounds interesting. It is also used by a project called "pixeltable" I have tested in the past. For testing and also for "one-shot processing" this could be reasonable. I do not yet get my head around the "embedded PostgresDB" for the specific Wayang Operator - but for testing, this could be beneficial. For me, embedding means, we bundle a part we need and which is not easily available - in case of Wayang Testing, I can envision an embedded PostrgesDB per Operator, so that we can verify the flow into and from the embedded DB. Is this what you are looking for? Cheers, Mirko Am Di., 21. Okt. 2025 um 18:39 Uhr schrieb Gavriilidis, Haralampos <[email protected]>: > Dear all, > > > I am currently preparing a PostgreSQL sink operator for Wayang. Is there > already a ticket for this, or should I create one? > > While working on it, I recalled that testing PostgreSQL-specific > functionality requires a separately managed PostgreSQL instance. For that > reason, I am currently exploring embedded PostgreSQL options such as > https://github.com/zonkyio/embedded-postgres and > https://github.com/opentable/otj-pg-embedded to simplify and enhance the > testing pipelines. Would this be something interesting for the project? > > > Best, > > Harry > -- Dr. Mirko Kämpf *Gründer & Coach * *maindset.ACADEMY* Müchelner Str. 23 06259 Frankleben
