Hi All, Currently the postgres database is up and running using Pablo suggestion ( Thanks Pablo!) But a new issue is in progress, we are trying to configure correctly postgres database to work with CDC ( change data capture) but any configuration doesn't work with readFromDebezium connector. Anyone had some experience with this? Using a postgres database with debezium on it. Postgres database is running in kubernetes pod. Thank you! Regards
On Wed, Jun 8, 2022 at 7:17 PM Pablo Estrada <[email protected]> wrote: > Hello Andoni, > thanks for bringing this to dev@. I took a quick look at other tests that > we have using Postgres. It seems that at least some of them use a > single-container Postgres database that is created using Kubernetes[1]. An > example test for JdbcIO [2], which runs on Dataflow, and uses the data that > it received passed in [3] to connect to postgres. > > This setup should work for this test. WDYT? We may just need to verify > that the postgres database supports CDC. > Best > -P. > > [1] > https://github.com/apache/beam/blob/master/.test-infra/kubernetes/postgres/postgres.yml > [2] > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy > [3] > https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java#L101-L124 > > On Wed, Jun 8, 2022 at 3:57 PM Andoni Guzman Becerra < > [email protected]> wrote: > >> Hi All, >> I have a question related to Cloud SQL. I'm adding a new load test , this >> test will work with debezium, checking that the readFromDebezium works >> well. For this I need to create a CloudSQL postgres instance to do some >> queries and check that Debezium catches the changes correctly. The >> problem is that CloudSQL instances do not accept new autorizathed >> connections for this constraint ( Restrict Authorized Networks on Cloud >> SQL instances) . Is there a way to connect to the instance? Or the >> constraints should be removed? >> The load test will run on a jenkins instance and the Cloud SQl will be >> created and deleted in each test. >> We have already tried several ways to do it. For example have the >> database in a container, Using a private ip to try to make the connections. >> All of them were unsuccessful. The main problem is that some runners >> like dataflow will run the tests and a connection with a database that can >> reach it's necessary >> Any help will be appreciated. >> >> Thank you! Regards >> >> -- >> >> Andoni Guzman | WIZELINE >> >> Software Engineer II >> >> [email protected] >> >> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal. >> >> >> >> >> >> >> >> >> *This email and its contents (including any attachments) are being sent >> toyou on the condition of confidentiality and may be protected by >> legalprivilege. Access to this email by anyone other than the intended >> recipientis unauthorized. If you are not the intended recipient, please >> immediatelynotify the sender by replying to this message and delete the >> materialimmediately from your system. Any further use, dissemination, >> distributionor reproduction of this email is strictly prohibited. Further, >> norepresentation is made with respect to any content contained in this >> email.* > > -- Andoni Guzman | WIZELINE Software Engineer II [email protected] Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal. -- *This email and its contents (including any attachments) are being sent to you on the condition of confidentiality and may be protected by legal privilege. Access to this email by anyone other than the intended recipient is unauthorized. If you are not the intended recipient, please immediately notify the sender by replying to this message and delete the material immediately from your system. Any further use, dissemination, distribution or reproduction of this email is strictly prohibited. Further, no representation is made with respect to any content contained in this email.*
