[
https://issues.apache.org/jira/browse/BEAM-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502336#comment-16502336
]
Raghu Angadi commented on BEAM-4485:
------------------------------------
Fortunately each of the two problems you mentioned have alternate ways:
# access on local launcher machine : KafkaIO needs this in-order to fetch
number of partitions for the topic. If you don't have access, then you can also
provide list of partitons explicity to reader. See `withTopicPartitions()` api.
Dataflow worker machine do need access.
# Difficulty with providing key files : This was a limitation in KafkaConsumer
configuration. Setting ACLs in Kafka got much better in 0.10.2
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients]
It allows setting these parameters in consumer config itself, which makes it
simpler to use with Dataflow. KafkaIO allows setting pretty much any of the
consumer config.
Does this help?
> Incredibly difficult to use KakfaIO + TLS + DataflowRunner
> ----------------------------------------------------------
>
> Key: BEAM-4485
> URL: https://issues.apache.org/jira/browse/BEAM-4485
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
> Reporter: Andre
> Assignee: Raghu Angadi
> Priority: Minor
> Fix For: Not applicable
>
>
> When attempting to use KafkaIO.Read with DataflowRunner, I have hit a lot of
> walls. The brokers need to be accessible both locally and from the dataflow
> runner instances. This means, when using TLS authentication, the
> keystore/truststore files need to be available locally and on the instances.
> I programmatically add the files to the pipeline options with
> {noformat}
> List<String> filesToStage =
> PipelineResources.detectClassPathResourcesToStage(IndicatorIngest.class.getClassLoader());
> filesToStage.add("trust.p12");
> filesToStage.add("server.p12");
> {noformat}
> but even when I do this, the remote file names are different. This means that
> I need to determine the remote file name myself, like this
> {noformat}
> PackageAttributes.forFileToStage(new File(filepath),
> filepath).getDestination().getName();
> {noformat}
> but that function is package private, so I need to wrap this call with a
> custom class in org.apache.beam.runners.dataflow.util. When I calculate this
> filename, I can use it to set the ssl.<thing>.location, but this is the wrong
> location locally, and it needs to be correct both locally and remotely. This
> means in my main I need to calculate the local files remote names, copy them
> to the local path with the same name, dynamically set the property to this
> path, and programmatically add these files to be staged so they hopefully
> have the same name on the worker. KafkaConsumer doesn't seem to provide any
> other way to specify where to get these keys from.
> My question is, I am supposed to be jumping through all these hoops, or am I
> doing something (or multiple things) completely wrong?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
