Hi Thomas, The issue occurs when the user does not have the READ permission on the consumer groups.
In DStreams group ID is configured in application, for example: https://github.com/gaborgsomogyi/spark-dstream-secure-kafka-app/blob/161bf02eb3677aac604d63499041f72231d0e371/src/main/scala/com/cloudera/spark/examples/DirectKafkaWordCount.scala#L59 In Strucuted Streaming the group ID is generated by Spark internally. Either one has to give access to "spark-kafka-source-*" group or in Spark 3.0 this prefix can be configured with "groupidprefix" parameter. BR, G On Wed, Feb 13, 2019 at 3:58 AM Allu👌🏽 Thomas <thomaspt...@icloud.com.invalid> wrote: > Hi There, > > My use case is to read a simple json message from Kafka queue using Spark > Structured Streaming. But I’m getting the following error message when I > run my Kafka consumer. I don’t get this error when using Spark direct > stream. The issue is happening only with structured streaming. Any help > would be greatly appreciated. > > > Exception in thread "main" > org.apache.spark.sql.streaming.StreamingQueryException: Not authorized to > access group: > spark-kafka-source-060f3ceb-09f4-4e28-8210-3ef8a845fc92--2038748645-driver-2 > === Streaming Query === > Identifier: [id = 6ab10eab-4f71-435c-8705-820e66cee47e, runId = > 48430367-9e14-450b-b8e0-27199b536403] > Current Committed Offsets: {} > Current Available Offsets: {} > > > Current State: ACTIVE > Thread State: RUNNABLE > > > Logical Plan: > KafkaSource[Subscribe[cla-claim-raw]] > at org.apache.spark.sql.execution.streaming.StreamExecution.org > $apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:295) > at > org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189) > Caused by: org.apache.kafka.common.errors.GroupAuthorizationException: Not > authorized to access group: > spark-kafka-source-060f3ceb-09f4-4e28-8210-3ef8a845fc92--2038748645-driver-2 > > Thanks, > Thomas Thomas >
