GitHub user Yuukadesu created a discussion: Implementing Integration E2E Tests for Java Client
# Proposal: Implementing Integration E2E Tests for Java Client ## Motivation To ensure the stability and reliability of the StreamPipes Java client library, this implementation adopts an end-to-end testing strategy aligned with the Go Client E2E. First, a test environment closely resembling production is set up, including the Backend, multiple Extension service instances, Kafka, CouchDB, and other required dependencies, so that test results are authentic and reproducible. Second, test cases are designed to cover two main objectives: **data path correctness** and **load balancing effectiveness**. They verify not only the end-to-end data flow (Kafka topic-in → adapter/source → processor → sink → Kafka topic-out), but also assert under the load of 70 pipelines and 70 adapters that elements do not all land on a single extension instance. To improve test efficiency and consistency, automated tests are written with JUnit 5, and connection parameters are injected via custom startup scripts and system properties. Finally, E2E tests are integrated into GitHub Actions CI: on every PR, images are built, the load-balanced environment is started, and Java Client E2E is executed, enabling continuous monitoring and timely feedback on the Java client and load-balancing behavior, and delivering a reliable and efficient Java client library for StreamPipes users. ## Goals - **Verify data path**: Ensure that adapters and pipelines created via the Java Client correctly complete the end-to-end flow (Kafka input → adapter → Boolean filter processor → Kafka output), and that filter semantics are validated. - **Verify load balancing**: At the scale of 70 resource groups (70 adapters + 70 pipelines), ensure that `selectedEndpointUrl` hits at least multiple extension instances (default ≥3), and that no single instance exceeds a configured share threshold (e.g. 40%), so that load is not concentrated on one instance. - **Early defect detection**: Through automated, repeatable E2E flows and CI integration, find and fix defects related to the Java Client API, Compact Adapter/Pipeline APIs, and load-balancing strategy as early as possible, reducing the cost of production fixes. - **Alignment with Go E2E style**: Execution flow is aligned with Go Client E2E (cleanup → create → start → verify → cleanup), making it easier to maintain and compare multi-language E2E tests later. ## Implementation The implementation ensures that the Java E2E test flow is consistent with the Go E2E test flow. A Docker-based environment is used with custom startup scripts that accept host, port, and other configuration parameters. E2E tests are triggered automatically on each PR via a GitHub Actions workflow and a script chain: the workflow builds images, starts the load-balanced stack with `docker-compose-loadbalancer.yml`, then runs `start-streampipes-client-e2e.sh -t java-client-e2e.sh`, which performs login, API key retrieval, and `mvn test` with the appropriate system properties; any failure fails the workflow so that code quality and load-balancing behavior are continuously monitored. GitHub link: https://github.com/apache/streampipes/discussions/4216 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
