Re: Kafka 0.10 with PySpark

2017-07-05 Thread Saisai Shao
Please see the reason in this thread ( https://github.com/apache/spark/pull/14340). It would better to use structured streaming instead. So I would like to -1 this patch. I think it's been a mistake to support > dstream in Python -- yes it satisfies a checkbox and Spark could claim > there's

Kafka 0.10 with PySpark

2017-07-04 Thread Daniel van der Ende
Hi, I'm working on integrating some pyspark code with Kafka. We'd like to use SSL/TLS, and so want to use Kafka 0.10. Because structured streaming is still marked alpha, we'd like to use Spark streaming. On this page, however, it indicates that the Kafka 0.10 integration in Spark does not support