If you have picked up all the changes for SPARK-18057, the Kafka “broker” supporting v1.0+ should be compatible with Spark's Kafka adapter.
Can you post more details about the “failed to send SSL close message” errors ? (The default Kafka version is 2.0.0 in Spark Kafka adapter after SPARK-18057 ) Thanks On Fri, Aug 17, 2018 at 3:53 PM Basil Hariri <basil.har...@microsoft.com.invalid> wrote: > Hi all, > > > > I work on Azure Event Hubs (Microsoft’s PaaS offering similar to Apache > Kafka) and am trying to get our new Kafka head > <https://azure.microsoft.com/en-us/blog/azure-event-hubs-for-kafka-ecosystems-in-public-preview/> > to play nice with Spark’s Kafka adapter. The goal is for our Kafka endpoint > to be completely compatible with Spark’s Kafka adapter, but I’m running > into some issues that I think are related to versioning. I’ve been trying > to tinker with the kafka-0-10-sql > <https://github.com/apache/spark/tree/master/external/kafka-0-10-sql> and > kafka-0-10 > <https://github.com/apache/spark/tree/master/external/kafka-0-10-sql> > adapters on Github and was wondering if someone could take a second to > point me in the right direction with: > > > > 1. What is the difference between those two adapters? My hunch is that > kafka-0-10-sql supports structured streaming while kafka-10-0 still uses > Spark streaming, but I haven’t found anything to verify that. > 2. Event Hubs’ Kafka endpoint only supports Kafka 1.0 and later, and > the errors I get when trying to connect to Spark (“failed to send SSL close > message” / broken pipe errors) have usually shown up when using Kafka v0.10 > applications with our endpoint. I built from source after I saw that both > libraries were updated for Kafka 2.0 support (late last week), but I’m > still running into the same issues. Do Spark’s Kafka adapters generally > downgrade to Kafka v0.10 protocols? If not, is there any other reason to > believe that a Kafka “broker” that doesn’t support v0.10 protocols but > supports v1.0+ would be incompatible with Spark’s Kafka adapter? > > > > Thanks in advance, please let me know if there’s a different place I > should be posting this > > > > Sincerely, > > Basil > > >
