unsubscribe

2021-09-20 Thread Sjoerd van Leent

Re: Problem with Kafka group.id

2020-03-24 Thread Sjoerd van Leent 
This is exactly the issue I am fighting against. Within a good number of 
organizations,  this is against policy. Another solution is necessary.


From: Spico Florin 
Sent: Tuesday, March 24, 2020 11:23:29 AM
To: Sethupathi T 
Cc: Sjoerd van Leent ; user@spark.apache.org 

Subject: Re: Problem with Kafka group.id

Hello!

Maybe you can find more information on the same issue reported here:
https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-KafkaSourceProvider.html<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjaceklaskowski.gitbooks.io%2Fspark-structured-streaming%2Fspark-sql-streaming-KafkaSourceProvider.html&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240026162&sdata=A1LHtb0I51ciZUnRq28FuBBjHzO6n3xg10enDWp8EnU%3D&reserved=0>

validateGeneralOptions makes sure that 
group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240036159&sdata=W%2FoB2aBKcr6URVcsH38QPhMF8YBAygqvFsbe9qML62c%3D&reserved=0>
 has not been specified and reports an IllegalArgumentException otherwise.

+

Kafka option 
'group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240036159&sdata=W%2FoB2aBKcr6URVcsH38QPhMF8YBAygqvFsbe9qML62c%3D&reserved=0>'
 is not supported as user-specified consumer groups are not used to track offset

https://github.com/Azure/azure-event-hubs-for-kafka/issues/35<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-event-hubs-for-kafka%2Fissues%2F35&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240046156&sdata=E8aih3f50LLrA9FVmjzr7AJoJ2%2FqAQoGMqMmqIzpt9k%3D&reserved=0>


I hope it helps, Florin

On Mon, Mar 23, 2020 at 5:45 PM Sethupathi T  
wrote:
I had exact same issue, the temp fix what I did was, took open source code from 
github, modified the 
group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240046156&sdata=ZCrnwgb8sd9%2BGFgJR0wPToVIdFCgsoYC%2BmIR2IYIDtw%3D&reserved=0>
 mandatory logic and built customized library.

Thanks,

On Tue, Mar 17, 2020 at 7:34 AM Sjoerd van Leent 
mailto:sjoerd.van.le...@alliander.com>> wrote:

Dear reader,



I must force the 
group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240056151&sdata=0%2BEbTnDZVzZmL8795qjAYK1%2FaqwYV0kPR3KxxGRLTBs%3D&reserved=0>
 of Kafka, as Kafka is under ACL control, however, doing so gives me the error:



Kafka option 
'group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240066146&sdata=nMX9V4IqdAhwLl5R5fHCa70z2wSSHhmCP%2FJv%2FxoHhCQ%3D&reserved=0>'
 is not supported as user-specified consumer groups are not used to track 
offsets.



This won’t work, as not being able to set it, basically disqualifies using 
Spark within our organization. How can I force (Py)Spark to respect the 
group.id<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgroup.id%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240066146&sdata=nMX9V4IqdAhwLl5R5fHCa70z2wSSHhmCP%2FJv%2FxoHhCQ%3D&reserved=0>
 used?



Met vriendelijke groet,



Sjoerd van Leent

Systeem Engineer | IT AST-B&R CSC


M   +31 6 11 24 52 27
Esjoerd.van.le...@alliander.com<mailto:sjoerd.van.le...@alliander.com>

Alliander N.V.  .  Postbus 50, 6920 AB Duiven, Nederland  .  Locatiecode: 
2PB2100  .  Utrechtseweg 68, 6812 AH 
Arnhem<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fmaps%2Fsearch%2FUtrechtseweg%2B68%2C%2B6812%2BAH%2BArnhem%3Fentry%3Dgmail%26source%3Dg&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240076145&sdata=aCrl%2FHJcbgkkyjnyj%2BOKRrKm3sj2s%2FJXo6hDDad2IvQ%3D&reserved=0>
  .  KvK 09104351 Arnhem  .  
www.alliander.com<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.alliander.com%2F&data=02%7C01%7C%7Ceddfd833591c4f16251708d7cfdd6cd6%7C697f104bd7cb48c8ac9fbd87105bafdc%7C0%7C0%7C637206422240076145&sdata=LdiVw61sn%2BRujuz2BajGrRiM9GEsexcC%2Bf5iTIhnUPA%3D&reserved=0&g

Problem with Kafka group.id

2020-03-17 Thread Sjoerd van Leent 
Dear reader,

I must force the group.id of Kafka, as Kafka is under ACL control, however, 
doing so gives me the error:

Kafka option 'group.id' is not supported as user-specified consumer groups are 
not used to track offsets.

This won't work, as not being able to set it, basically disqualifies using 
Spark within our organization. How can I force (Py)Spark to respect the 
group.id used?

Met vriendelijke groet,

Sjoerd van Leent
Systeem Engineer | IT AST-B&R CSC


M   +31 6 11 24 52 27
Esjoerd.van.le...@alliander.com<mailto:sjoerd.van.le...@alliander.com>

Alliander N.V.  .  Postbus 50, 6920 AB Duiven, Nederland  .  Locatiecode: 
2PB2100  .  Utrechtseweg 68, 6812 AH Arnhem  .  KvK 09104351 Arnhem  .  
www.alliander.com<http://www.alliander.com/>

De inhoud van deze e-mail, inclusief bijlagen, is persoonlijk en vertrouwelijk. 
Mocht dit bericht niet voor u bedoeld zijn, informeer dan per omgaande de 
afzender en verwijder dit bericht. Gelieve deze e-mail, inclusief eventuele 
bijlagen, niet te gebruiken, kopiëren of door te sturen aan derden.