Hello Beam community,
I am wondering whether it is possible to connect to a secure Kafka broker
with the python beam SDK. So for example, in the code below, how would I
make the "ssl.truststore.location" and "ssl.keystore.location" accessible
inside the Java SDK harness which runs the code.
ReadFromKafka(
consumer_config={
"bootstrap.servers": "bootstrap-server:17032",
"security.protocol": "SSL",
"ssl.truststore.location":
"/opt/keys/client.truststore.jks", # how do I make this available to
the Java SDK harness
"ssl.truststore.password": "password",
"ssl.keystore.type": "PKCS12",
"ssl.keystore.location": "/opt/keys/client.keystore.p12",
# how do I make this available to the Java SDK harness
"ssl.keystore.password": "password",
"group.id": "group",
"basic.auth.credentials.source": "USER_INFO",
"schema.registry.basic.auth.user.info": "user:password"
},
topics=["topic"],
max_num_records=2,
# expansion_service="localhost:56938"
)
I tried building a custom image that has pulls credentials into the
container based on "apache/beam_java11_sdk:2.27.0" and using the
"--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}"
argument, but it does not appear to be used when the pipeline runs on Flink.
Thank you for the great tool and any help or advice is greatly appreciated.
--
paul
