Hi All, Please find the updated User Story document here <https://docs.google.com/document/d/1LBHz-kstnFpSTxXk1ouPHy7FD8Fd6MKivkXIj2QMd-g/edit?usp=sharing> . I also started with implementation of the WSO2 SP io-extension for this feature. Please find the github repo here <https://github.com/chathuranga95/siddhi-io-cdc>. Note that currently WIP.
Best regards, Chathuranga Siriwardhana, Software Engineering Intern. Mobile: +94713604485 On Mon, Aug 20, 2018 at 4:26 PM Chathuranga Siriwardhana < chathuran...@wso2.com> wrote: > Hi all, > > I tried the CDC for MySQL with Debezium[1] and implemented as a simple > java program using embedded debezium[2]. > Currently I am trying the Embedded Debezium for Oracle. The respected java > code can be found here. > > > https://github.com/chathuranga95/Embedded-Debezium-CDC/blob/master/src/main/java/io/debezium/App.java > > Please note that the runMysqlCDC() is returning the MySQL database change > event details with row data and currently runOracleCDC() is not producing > expected output. Find the details of configuration from [2]. > > I also got started with the user story for this functionality. The source > will be as follows for MySQL bin-log CDC. > > > @source(type=’cdc’ , url=’jdbc:mysql://root:root@localhost > :3306/dbStudents’, > > @map(type='json', fail.on.missing.attribute='false' > ,@attributes(operation=’$.operation, beforeID=’$.before.id’, > beforeName=’$.before.name’, afterID=’$.after.id’, afterName=’$.after.name > ’))) > > define stream cdcEvent(operation String, beforeID String, beforeName > String, afterID String, afterName String); > > [1] http://debezium.io/ > [2] http://debezium.io/docs/embedded/ > > Best regards, > Chathuranga Siriwardhana, > Software Engineering Intern. > Mobile: +94713604485 > > > On Tue, Jul 24, 2018 at 6:38 PM Niveathika Rajendran <niveath...@wso2.com> > wrote: > >> Hi Chathuranga, >> >> As discussed offline let's look at Debizum[1] as well especially >> embedded Debizium[2], which removes the dependency on Kafka[3] >> >> Also, shall we have a User Story review for the above functionality? >> >> [1] http://debezium.io/ >> [2] http://debezium.io/docs/embedded/ >> [3] Mail Thread : CDC (Change Data Capture) for Product-SP >> >> Best Regards, >> *Niveathika Rajendran,* >> *Software Engineer.* >> *Mobile : +94 077 903 7536* >> >> >> >> >> >> On Tue, Jul 24, 2018 at 12:25 PM Chathuranga Siriwardhana < >> chathuran...@wso2.com> wrote: >> >>> >>> >>> >>> >>> >>> >>> >>> *Hi all,I am currently working on the 'Change Data Capture for WSO2SP'. >>> As the first part of the project, I have to capture the change >>> events(INSERT, UPDATE, DELETE mainly) of a few databases using java. Mainly >>> I have focused on MySQL, MS-SQL server, ORACLE and PostgreSQL databases.For >>> now, I have used the following approaches to try to capture the change data >>> of relevant databases.DatabaseMethod UsedUsed connectors / >>> APIs.OutcomeStatusCommentsReferencesMySQLBinlog file based java event >>> listening. (Asynchronous)Mysql-binlog-connector-java(Apache 2.0 >>> License)Change events(INSERT, UPDATE, DELETE) can be captured along with >>> the relevant row data.Complete[1]OracleDBDatabase Change Notification >>> feature based java event listeningOracle jdbc driverChange events(INSERT, >>> UPDATE, DELETE) can be captured without row data.Partially >>> Complete[2]PostgreSQLListen Notify functionality of the Postgres with >>> table-wise configuration and polling in the program.Postgres jdbc >>> driverChange events(INSERT, UPDATE, DELETE) without row data.IncompleteThis >>> method was discarded because a trigger should be written into each table in >>> order to get change events from that table.[3], [4]PostgreSQLReplication >>> Data based java pollingPhysical and Logical replication APIChange >>> events(INSERT, UPDATE, DELETE) can be captured. Row data is not returning >>> for DELETE event.Partially Complete[5]MS-SQL ServerTransaction Log based >>> java pollingMs-sql jdbc connectorOn progress; Transaction log is >>> read.On-going.[6], [7][1] >>> https://github.com/shyiko/mysql-binlog-connector-java >>> <https://github.com/shyiko/mysql-binlog-connector-java>[2] >>> https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820 >>> <https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820>[3] >>> https://www.postgresql.org/docs/current/static/sql-notify.html >>> <https://www.postgresql.org/docs/current/static/sql-notify.html>[4] >>> https://www.postgresql.org/docs/current/static/sql-listen.html >>> <https://www.postgresql.org/docs/current/static/sql-listen.html>[5] >>> https://jdbc.postgresql.org/documentation/head/replication.html >>> <https://jdbc.postgresql.org/documentation/head/replication.html>[6]https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-2017 >>> <https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-2017>[7] >>> https://dzone.com/articles/how-use-sql-server-transaction >>> <https://dzone.com/articles/how-use-sql-server-transaction>*Please find >>> the shared Google Sheet containing above table. You may comment about the >>> existing approaches / mention any new approaches. >>> >>> Best regards, >>> Chathuranga Siriwardhana, >>> Software Engineering Intern. >>> Mobile: +94713604485 >>> >>
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
