Add E2E test for Kafka to Pub/Sub complete example
Hi Beam Community, Recently our team added Kafka to Pub/Sub example pipeline to Apache Beam repository. We decided to move E2E tests out of the scope because we needed more time for investigation and implementation. And now the test is ready, the PR is open. Here are the jira tickets for this PR: https://issues.apache.org/jira/browse/BEAM-11410 https://issues.apache.org/jira/browse/BEAM-11411 Please share your feedback/comments about this proposal in the thread. Here is the PR, please take a look at it and feel free to share your comments: https://github.com/apache/beam/pull/13636 Thank you, Ramazan
Problems with E2E test
Hi Beam!
We've been writing E2E test for KafkaToPubsub example pipeline. Instead of
depending on some real
Cloud Pubsub and Kafka instances we decided to use Testcontainers.
We launch Kafka and PubSub Emulator containers and after that we pass
containers urls into pipeline options and run the pipeline.
During PR review we received a request for turning this test into IT so it
would run in Dataflow Runner
instead of Direct Runner.
Trying to do so, we've ran into some troubles with that:
1. While running the test all Docker containers start at the machine where the
test is running,
so in order for this test to work properly dataflow job should be able to
reach test-runner machine by a public IP.
I certainly can't do it on my local machine, not sure how it will behave
when running in CI environment.
2. When we pass our fake PubSub url into the dataflow job we receive following
error:
json
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "(f214233f9dbe6968): The workflow could not be created. Causes:
(f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub URL.",
"reason" : "badRequest"
} ],
"message" : "(f214233f9dbe6968): The workflow could not be created. Causes:
(f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub URL.",
"status" : "INVALID_ARGUMENT"
}
Not sure how this can be avoided, looks like the job will only accept the real
Cloud PubSub url.
It would be great if you share some thoughts or any suggestions how it can be
solved!
Migrating Gradle scripts to Kotlin
Hi Beam Dev! I started to work on this issue https://issues.apache.org/jira/browse/BEAM-11589 - migrating Gradle scripts from Groovy to Kotlin. Currently, I've migrated settings.gradle, buildSrc/build.gradle and release/build.gradle files to Kotlin. I believe that this work should be done incrementally, so I wanted to ask if this amount of changes is enough for a PR or should I migrate more modules?
Re: Migrating Gradle scripts to Kotlin
Yes, that's me, thank you! From: Kenneth Knowles Sent: Tuesday, February 2, 2021 9:23:45 PM To: dev Subject: Re: Migrating Gradle scripts to Kotlin I took a guess that the Jira user 'ramazan-yapparov' was you and have added you to the project and assigned the ticket. Let me know if this is not you. Kenn On Thu, Jan 28, 2021 at 2:26 PM Brian Hulette mailto:bhule...@google.com>> wrote: I think generally smaller PRs are preferable if the work is easily separable, which it sounds like this is. I'd recommend you go ahead and put up a PR, you might even split it into one PR per file if that makes sense to you. Side note - do you have a Jira ID? We should get BEAM-11589 assigned to you if you are working on it :) Brian On Thu, Jan 28, 2021 at 2:52 AM Ramazan Yapparov mailto:ramazan.yappa...@akvelon.com>> wrote: Hi Beam Dev! I started to work on this issue https://issues.apache.org/jira/browse/BEAM-11589 - migrating Gradle scripts from Groovy to Kotlin. Currently, I've migrated settings.gradle, buildSrc/build.gradle and release/build.gradle files to Kotlin. I believe that this work should be done incrementally, so I wanted to ask if this amount of changes is enough for a PR or should I migrate more modules?
