Re: PySpark Direct Streaming : SASL Security Compatibility Issue

2018-11-28 Thread Gabor Somogyi
Hi Muthuraman, Previous to 0.9, Kafka had no built-in security features but DStreams support only kafka 0.8. Suggest to use structured streaming where 0.10+ (2.0.0 in spark 2.4) support is available. BR, G On Wed, Nov 28, 2018 at 4:15 AM Ramaswamy, Muthuraman < muthuraman.ramasw...@viasat.com>

PySpark Direct Streaming : SASL Security Compatibility Issue

2018-11-27 Thread Ramaswamy, Muthuraman
Hi All, I am using PySpark Direct Streaming to connect to a remote secured Kafka broker and is secured with Kerberos Authentication. The KafkaUtils.createDirectStream python call gives me the following error: 18/11/27 18:20:05 WARN VerifiableProperties: Property sasl.mechanism is not valid 18/