Hi Beam!
We've been writing E2E test for KafkaToPubsub example pipeline. Instead of
depending on some real
Cloud Pubsub and Kafka instances we decided to use Testcontainers.
We launch Kafka and PubSub Emulator containers and after that we pass
containers urls into pipeline options and run the pipeline.
During PR review we received a request for turning this test into IT so it
would run in Dataflow Runner
instead of Direct Runner.
Trying to do so, we've ran into some troubles with that:
1. While running the test all Docker containers start at the machine where the
test is running,
so in order for this test to work properly dataflow job should be able to
reach test-runner machine by a public IP.
I certainly can't do it on my local machine, not sure how it will behave
when running in CI environment.
2. When we pass our fake PubSub url into the dataflow job we receive following
error:
json
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "(f214233f9dbe6968): The workflow could not be created. Causes:
(f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub URL.",
"reason" : "badRequest"
} ],
"message" : "(f214233f9dbe6968): The workflow could not be created. Causes:
(f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub URL.",
"status" : "INVALID_ARGUMENT"
}
Not sure how this can be avoided, looks like the job will only accept the real
Cloud PubSub url.
It would be great if you share some thoughts or any suggestions how it can be
solved!
