This is exactly the issue I am fighting against. Within a good number of organizations, this is against policy. Another solution is necessary.
________________________________ From: Spico Florin <spicoflo...@gmail.com> Sent: Tuesday, March 24, 2020 11:23:29 AM To: Sethupathi T <sethu....@googlemail.com.invalid> Cc: Sjoerd van Leent <sjoerd.van.le...@alliander.com>; user@spark.apache.org <user@spark.apache.org> Subject: Re: Problem with Kafka group.id Hello! Maybe you can find more information on the same issue reported here: https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-KafkaSourceProvider.html<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjaceklaskowski.gitbooks.io%2Fspark-structured-streaming%2Fspark-sql-streaming-KafkaSourceProvider.html&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240026162&sdata=A1LHtb0I51ciZUnRq28FuBBjHzO6n3xg10enDWp8EnU%3D&reserved=0> validateGeneralOptions makes sure that group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240036159&sdata=W%2FoB2aBKcr6URVcsH38QPhMF8YBAygqvFsbe9qML62c%3D&reserved=0> has not been specified and reports an IllegalArgumentException otherwise. + Kafka option 'group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240036159&sdata=W%2FoB2aBKcr6URVcsH38QPhMF8YBAygqvFsbe9qML62c%3D&reserved=0>' is not supported as user-specified consumer groups are not used to track offset https://github.com/Azure/azure-event-hubs-for-kafka/issues/35<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-event-hubs-for-kafka%2Fissues%2F35&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240046156&sdata=E8aih3f50LLrA9FVmjzr7AJoJ2%2FqAQoGMqMmqIzpt9k%3D&reserved=0> I hope it helps, Florin On Mon, Mar 23, 2020 at 5:45 PM Sethupathi T <sethu....@googlemail.com.invalid> wrote: I had exact same issue, the temp fix what I did was, took open source code from github, modified the group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240046156&sdata=ZCrnwgb8sd9%2BGFgJR0wPToVIdFCgsoYC%2BmIR2IYIDtw%3D&reserved=0> mandatory logic and built customized library. Thanks, On Tue, Mar 17, 2020 at 7:34 AM Sjoerd van Leent <sjoerd.van.le...@alliander.com<mailto:sjoerd.van.le...@alliander.com>> wrote: Dear reader, I must force the group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240056151&sdata=0%2BEbTnDZVzZmL8795qjAYK1%2FaqwYV0kPR3KxxGRLTBs%3D&reserved=0> of Kafka, as Kafka is under ACL control, however, doing so gives me the error: Kafka option 'group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240066146&sdata=nMX9V4IqdAhwLl5R5fHCa70z2wSSHhmCP%2FJv%2FxoHhCQ%3D&reserved=0>' is not supported as user-specified consumer groups are not used to track offsets. This won’t work, as not being able to set it, basically disqualifies using Spark within our organization. How can I force (Py)Spark to respect the group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240066146&sdata=nMX9V4IqdAhwLl5R5fHCa70z2wSSHhmCP%2FJv%2FxoHhCQ%3D&reserved=0> used? Met vriendelijke groet, Sjoerd van Leent Systeem Engineer | IT AST-B&R CSC M +31 6 11 24 52 27 E sjoerd.van.le...@alliander.com<mailto:sjoerd.van.le...@alliander.com> Alliander N.V. . Postbus 50, 6920 AB Duiven, Nederland . Locatiecode: 2PB2100 . Utrechtseweg 68, 6812 AH Arnhem<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fmaps%2Fsearch%2FUtrechtseweg%2B68%2C%2B6812%2BAH%2BArnhem%3Fentry%3Dgmail%26source%3Dg&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240076145&sdata=aCrl%2FHJcbgkkyjnyj%2BOKRrKm3sj2s%2FJXo6hDDad2IvQ%3D&reserved=0> . KvK 09104351 Arnhem . www.alliander.com<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.alliander.com%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240076145&sdata=LdiVw61sn%2BRujuz2BajGrRiM9GEsexcC%2Bf5iTIhnUPA%3D&reserved=0> De inhoud van deze e-mail, inclusief bijlagen, is persoonlijk en vertrouwelijk. Mocht dit bericht niet voor u bedoeld zijn, informeer dan per omgaande de afzender en verwijder dit bericht. Gelieve deze e-mail, inclusief eventuele bijlagen, niet te gebruiken, kopiëren of door te sturen aan derden.
