[ https://issues.apache.org/jira/browse/BEAM-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513629#comment-15513629 ]
ASF GitHub Bot commented on BEAM-549: ------------------------------------- Github user amitsela closed the pull request at: https://github.com/apache/incubator-beam/pull/822 > SparkRunner should support Beam's KafkaIO instead of providing it's own. > ------------------------------------------------------------------------ > > Key: BEAM-549 > URL: https://issues.apache.org/jira/browse/BEAM-549 > Project: Beam > Issue Type: Bug > Components: runner-spark > Reporter: Amit Sela > Assignee: Amit Sela > > For portability, and in the spirit of Apache Beam, the SparkRunner should use > the Beam implementation of KafkaIO instead of it's own. > Having said that, the runner will translate the KafkaIO as defined in the > pipeline into it's own internal implementation, but should still map the > properties the user defined in the pipeline in a way that the IO behaves the > same - i.e., brokers, topic, etc. > Eventually, the SparkRunner will implement reading from Kafka using Spark's > KafakUtils.createDirectStream() as described here: > http://spark.apache.org/docs/1.6.2/streaming-kafka-integration.html#approach-2-direct-approach-no-receivers > -- This message was sent by Atlassian JIRA (v6.3.4#6332)