Hi all, Thanks for the responses. Regarding Kyle Weaver's suggestion, I tried a combination of "--environment_type=DOCKER" and '--environment_config=beam_python_sdk:0.0.1' where beam_python_sdk:0.0.1 is an image I built. That made it possible to override the python sdk harness, but I was wondering if there is something similar for overriding the Java sdk harness?
Thanks again. On Tue, Feb 2, 2021 at 9:08 PM Kyle Weaver <[email protected]> wrote: > AFAIK sdk_harness_container_image_overrides only works for the Dataflow > runner. For other runners I think you will have to change the default > environment config: > https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L53-L71 > > On Tue, Feb 2, 2021 at 11:56 AM Brian Hulette <[email protected]> wrote: > >> Hm I would expect that to work. Can you tell what container Flink is >> using if it's not using the one you specified? +Chamikara Jayalath >> <[email protected]> may have some insight here >> >> Brian >> >> On Tue, Feb 2, 2021 at 3:27 AM Paul Nimusiima <[email protected]> >> wrote: >> >>> Hello Beam community, >>> >>> I am wondering whether it is possible to connect to a secure Kafka >>> broker with the python beam SDK. So for example, in the code below, how >>> would I make the "ssl.truststore.location" and "ssl.keystore.location" >>> accessible inside the Java SDK harness which runs the code. >>> >>> ReadFromKafka( >>> consumer_config={ >>> "bootstrap.servers": "bootstrap-server:17032", >>> "security.protocol": "SSL", >>> "ssl.truststore.location": "/opt/keys/client.truststore.jks", # >>> how do I make this available to the Java SDK harness >>> "ssl.truststore.password": "password", >>> "ssl.keystore.type": "PKCS12", >>> "ssl.keystore.location": "/opt/keys/client.keystore.p12", # how >>> do I make this available to the Java SDK harness >>> "ssl.keystore.password": "password", >>> "group.id": "group", >>> "basic.auth.credentials.source": "USER_INFO", >>> "schema.registry.basic.auth.user.info": "user:password" >>> }, >>> topics=["topic"], >>> max_num_records=2, >>> # expansion_service="localhost:56938" >>> ) >>> >>> I tried building a custom image that has pulls credentials into the >>> container based on "apache/beam_java11_sdk:2.27.0" and using the >>> "--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}" >>> argument, but it does not appear to be used when the pipeline runs on >>> Flink. >>> >>> Thank you for the great tool and any help or advice is greatly >>> appreciated. >>> >>> -- >>> paul >>> >> -- paul
