Hello! Maybe you can find more information on the same issue reported here: https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-KafkaSourceProvider.html
validateGeneralOptions makes sure that group.id has not been specified and reports an IllegalArgumentException otherwise. + Kafka option 'group.id' is not supported as user-specified consumer groups are not used to track offset https://github.com/Azure/azure-event-hubs-for-kafka/issues/35 I hope it helps, Florin On Mon, Mar 23, 2020 at 5:45 PM Sethupathi T <sethu....@googlemail.com.invalid> wrote: > I had exact same issue, the temp fix what I did was, took open source code > from github, modified the group.id mandatory logic and built customized > library. > > Thanks, > > On Tue, Mar 17, 2020 at 7:34 AM Sjoerd van Leent < > sjoerd.van.le...@alliander.com> wrote: > >> Dear reader, >> >> >> >> I must force the group.id of Kafka, as Kafka is under ACL control, >> however, doing so gives me the error: >> >> >> >> Kafka option 'group.id' is not supported as user-specified consumer >> groups are not used to track offsets. >> >> >> >> This won’t work, as not being able to set it, basically disqualifies >> using Spark within our organization. How can I force (Py)Spark to respect >> the group.id used? >> >> >> >> Met vriendelijke groet, >> >> >> >> *Sjoerd van Leent* >> >> Systeem Engineer | IT AST-B&R CSC >> >> >> >> *M *+31 6 11 24 52 27 >> *E * sjoerd.van.le...@alliander.com >> >> >> *Alliander N.V. *. Postbus 50, 6920 AB Duiven, Nederland . >> Locatiecode: 2PB2100 . Utrechtseweg 68, 6812 AH Arnhem >> <https://www.google.com/maps/search/Utrechtseweg+68,+6812+AH+Arnhem?entry=gmail&source=g> >> . KvK 09104351 Arnhem . *www.alliander.com >> <http://www.alliander.com/> * >> >> >> >> De inhoud van deze e-mail, inclusief bijlagen, is persoonlijk en >> vertrouwelijk. Mocht dit bericht niet voor u bedoeld zijn, informeer dan >> per omgaande de afzender en verwijder dit bericht. Gelieve deze e-mail, >> inclusief eventuele bijlagen, niet te gebruiken, kopiëren of door te sturen >> aan derden. >> >> >> >> >> >
