[jira] [Updated] (BEAM-9525) KafkaIO attempting to make connection when uploading via Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9525: --- Status: Open (was: Triage Needed) > KafkaIO attempting to make connection when uploading via Dataflow Runner > > > Key: BEAM-9525 > URL: https://issues.apache.org/jira/browse/BEAM-9525 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Reporter: Ross Rothenstine >Priority: Minor > > Given our Kafka is running in GCP, within a VPC that is not accessible > outside of the VPC. > When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD > worker, it attempts to make a connection to the broker. Our broker isn't > available outside the VPC, so this process fails. We can remedy this by > running a pod from within the VPC to upload, but it causes an awkward step > within our CI/CD process. > If there is a more elegant solution to this, such as KafkaIO taking in a > ValueProvider for bootstrap servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9525) KafkaIO attempting to make connection when uploading via Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9525: --- Status: Triage Needed (was: Open) > KafkaIO attempting to make connection when uploading via Dataflow Runner > > > Key: BEAM-9525 > URL: https://issues.apache.org/jira/browse/BEAM-9525 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Reporter: Ross Rothenstine >Priority: Minor > > Given our Kafka is running in GCP, within a VPC that is not accessible > outside of the VPC. > When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD > worker, it attempts to make a connection to the broker. Our broker isn't > available outside the VPC, so this process fails. We can remedy this by > running a pod from within the VPC to upload, but it causes an awkward step > within our CI/CD process. > If there is a more elegant solution to this, such as KafkaIO taking in a > ValueProvider for bootstrap servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9525) KafkaIO attempting to make connection when uploading via Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9525: --- Status: Open (was: Triage Needed) > KafkaIO attempting to make connection when uploading via Dataflow Runner > > > Key: BEAM-9525 > URL: https://issues.apache.org/jira/browse/BEAM-9525 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Reporter: Ross Rothenstine >Priority: Minor > > Given our Kafka is running in GCP, within a VPC that is not accessible > outside of the VPC. > When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD > worker, it attempts to make a connection to the broker. Our broker isn't > available outside the VPC, so this process fails. We can remedy this by > running a pod from within the VPC to upload, but it causes an awkward step > within our CI/CD process. > If there is a more elegant solution to this, such as KafkaIO taking in a > ValueProvider for bootstrap servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
