[jira] [Commented] (BEAM-9627) KafkaIO needs better support for SSL
[
https://issues.apache.org/jira/browse/BEAM-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220754#comment-17220754
]
Alexey Romanenko commented on BEAM-9627:
Hi [~mwz] , I don't think it's possible to use from Python API for the moment
since Python SDK version of Kafka IO uses Java {{KafkaIO}} through external
PTransform which is exposed by \{{KafkaIO.Read.External}} class and it's
\{{Configuration}} and it uses default \{{consumerFactoryFn}}. So yes, please
open a new issue about that for \{{io-py-kafka}} component.
> KafkaIO needs better support for SSL
>
>
> Key: BEAM-9627
> URL: https://issues.apache.org/jira/browse/BEAM-9627
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
>Reporter: Daniel Mills
>Priority: P3
> Fix For: Not applicable
>
>
> Configuring SSL for kafka requires pointing an option at local files
> containing keys and roots of trust as described here:
> [https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/]
> Currently, it is somewhat tricky to ensure that these files are written
> before KafkaIO starts reading from the source; one potential option would be
> to add an init hook where the user could download keys from the keystore of
> their choice and write them to local files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9627) KafkaIO needs better support for SSL
[ https://issues.apache.org/jira/browse/BEAM-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218336#comment-17218336 ] Michael Wizner commented on BEAM-9627: -- Hi [~aromanenko], is this solution applicable to the Python API? I've been wondering how to do this in Python since Python API doesn't expose those factory functions. Is this currently possible to do via Python API or should I raise a new issue about it in the io-py-kafka component? > KafkaIO needs better support for SSL > > > Key: BEAM-9627 > URL: https://issues.apache.org/jira/browse/BEAM-9627 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Daniel Mills >Priority: P3 > Fix For: Not applicable > > > Configuring SSL for kafka requires pointing an option at local files > containing keys and roots of trust as described here: > [https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/] > Currently, it is somewhat tricky to ensure that these files are written > before KafkaIO starts reading from the source; one potential option would be > to add an init hook where the user could download keys from the keystore of > their choice and write them to local files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9627) KafkaIO needs better support for SSL
[
https://issues.apache.org/jira/browse/BEAM-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072966#comment-17072966
]
Alexey Romanenko commented on BEAM-9627:
Actually, this hook already exists. You can try to use
{{KafkaIO.Read.withConsumerFactoryFn()}} and implement custom consumer
function, which will be called to create Kafka consumer on worker. In this
function you can copy your files to local temp directories from whatever place
you need.
Some more concrete examples about that approach:
https://stackoverflow.com/questions/42726011/truststore-and-google-cloud-dataflow/53549757
https://stackoverflow.com/questions/54337653/connect-to-kafka-with-ssl-using-kafkaio-on-google-dataflow
> KafkaIO needs better support for SSL
>
>
> Key: BEAM-9627
> URL: https://issues.apache.org/jira/browse/BEAM-9627
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
>Reporter: Daniel Mills
>Priority: Minor
>
> Configuring SSL for kafka requires pointing an option at local files
> containing keys and roots of trust as described here:
> [https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/]
> Currently, it is somewhat tricky to ensure that these files are written
> before KafkaIO starts reading from the source; one potential option would be
> to add an init hook where the user could download keys from the keystore of
> their choice and write them to local files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
