[ https://issues.apache.org/jira/browse/BEAM-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072936#comment-17072936 ]
Alexey Romanenko commented on BEAM-9525: ---------------------------------------- Do you know what kind of request it sends to broker in the beginning? I can assume that it tries to fetch a list of topic(s) partitions (the source splits that are used then to distribute a work across workers). In this case you try to use {{withTopicPartitions()}} and provide a list of partitions. In this case it won't fetch it from broker. > KafkaIO attempting to make connection when uploading via Dataflow Runner > ------------------------------------------------------------------------ > > Key: BEAM-9525 > URL: https://issues.apache.org/jira/browse/BEAM-9525 > Project: Beam > Issue Type: Bug > Components: io-java-kafka > Reporter: Ross Rothenstine > Priority: Minor > > Given our Kafka is running in GCP, within a VPC that is not accessible > outside of the VPC. > When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD > worker, it attempts to make a connection to the broker. Our broker isn't > available outside the VPC, so this process fails. We can remedy this by > running a pod from within the VPC to upload, but it causes an awkward step > within our CI/CD process. > If there is a more elegant solution to this, such as KafkaIO taking in a > ValueProvider for bootstrap servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)