Re: Flink SQL: MySQL to Elaticsearch soft delete

2023-10-22 Thread Feng Jin
Hi Hemi You can not just filter the delete records. You must use the following syntax to generate a delete record. ``` CREATE TABLE test_source (f1 xxx, f2, xxx, f3 xxx, deleted boolean) with (.); INSERT INTO es_sink SELECT f1, f2, f3 FROM ( SELECT *, ROW_NUMBER() OVER (PARTITION BY f1

Re: Flink SQL: MySQL to Elaticsearch soft delete

2023-10-21 Thread Feng Jin
Hi Hemi, One possible way, but it may generate many useless states. As shown below: ``` CREATE TABLE test_source (f1 xxx, f2, xxx, f3 xxx, deleted boolean) with (.); INSERT INTO es_sink SELECT f1, f2, f3 FROM ( SELECT *, ROW_NUMBER() OVER (PARTITION BY f1, f2 ORDER BY proctime()) as

Flink SQL: MySQL to Elaticsearch soft delete

2023-10-19 Thread Hemi Grs
hello everyone, right now I'm using flink to sync from mysql to elasticsearch and so far so good. If we insert, update, or delete it will sync from mysql to elastic without any problem. The problem I have right now is the application is not actually doing hard delete to the records in mysql, but