Please, see my comments inline. > On 12 Jan 2022, at 20:15, Marco Robles <[email protected]> wrote: > > I am looking for integration tests for SDF implementation (testing the > pipeline) for unbounded-stream that may help me as an example to validate > some expected behaviors in my current implementation for the SDF tests for > PulsarIO connector.
Are you looking for something specific to test? How should this be different from other integration tests? > I was using KafkaIO as an example for the testing, but it seems the SDF > version for the testing is not being used (Pipeline tests > <https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java>). > Actually, both types of Kafka read, SDF or Source API one, can be used within KafkaIO.Read (which is used in KafkaIOIT.testKafkaIOReadsAndWritesCorrectlyInStreaming() to read messages in testing pipeline). Depending on provided options, KafkaIO.Read will expand either into ReadFromKafkaViaUnbounded or ReadFromKafkaViaSDF transforms [1] So, you need to make sure that you run a test with proper options and it executes the right code branch in the end. [1] https://github.com/apache/beam/blob/a3bf36eae08a7e31ec862674180466166b9cf501/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1294 — Alexey > > Thanks in advance > > -- > Marco Robles | WIZELINE > Software Engineer > [email protected] > <mailto:[email protected]> > This email and its contents (including any attachments) are being sent to > you on the condition of confidentiality and may be protected by legal > privilege. Access to this email by anyone other than the intended recipient > is unauthorized. If you are not the intended recipient, please immediately > notify the sender by replying to this message and delete the material > immediately from your system. Any further use, dissemination, distribution > or reproduction of this email is strictly prohibited. Further, no > representation is made with respect to any content contained in this email.
