This e2e test only need test whether the job is successful and the result data is ok. If the active triggering failure function is working fine and the result is expected we can think that state recovery is normal.
*How to implement the test logic of all engines uniformly ? * *I need some time to think.* Zongwen Li <[email protected]> 于2022年9月20日周二 21:01写道： > It is necessary to test whether the state recovery of the connector is > correct, the problem is how to implement the test logic of all engines > uniformly. > > Best, > Zongwen Li > > JUN GAO <[email protected]> 于2022年9月20日周二 13:37写道： > > > Description > > > > We need support data consistency test in connector v2 e2e. I have some > idea > > about it and welcome everyone to discuss. > > Test Sink ConnectorFake Source Connector > > > > If we want to test the data consistency of a sink connector, We can > > use the Fake > > Source connector. The Fake Source Connector support define row numbers > and > > Primary key fields in the feature. Defile Primary key fields is useful to > > test exactly-once sink which implement exactly-once by Idempotent write > > data. If we can simulate task failure and then restore task, We can > > complete the data consistency test. > > How to simulate task failure and restore task. > > > > I think we can use Fake Source connector to simulate task failure too. We > > can add some active triggering failure function in Fake Source. To ensure > > Fake Source can support read playback, the Fake Source need support > > snapshot too. > > How to check data > > > > We can check the rows that wrote in sink. > > Test Source ConnectorTest JDBC Sink Connector > > > > If we want to test the data consistency of a source connector, We can > > add a Test > > JDBC Sink connector. It need support exactly-once. > > How to simulate task failure and restore task. > > > > We can add some active triggering failure function in Test JDBC Sink > > connector. > > How to check data > > > > There are two ways to do it. > > > > First one: After the job xxxSource -> TestJDBCSink finished, we can > > automatically create a job JDBCSource -> AssertSink and use AssertSink to > > check data. > > > > shortcoming This way need run two jobs. > > > > advantage This way can do test standardization, people only need config > the > > check rules in AssertSink connector. > > > > The second one is add a java program to check data in MySQL/PG. > > > > shortcoming This way can not do test standardization, every source > > connector e2e need add the check program and define the check rules > > themselves. > > > > advantage only need run one job. > > > > -- > > > > Best Regards > > > > ------------ > > > > EricJoy2048 > > [email protected] > > > -- Best Regards ------------ EricJoy2048 [email protected]
