Re: [Structured Streaming] Kafka group.id is fixed

2018-11-19 Thread Cody Koeninger
Anastasios it looks like you already identified the two lines that need to change, the string interpolation that depends on UUID.randomUUID and metadataPath.hashCode. I'd factor that out into a function that returns the group id. That function would also need to take the "parameters" variable

Re: [Structured Streaming] Kafka group.id is fixed

2018-11-19 Thread Anastasios Zouzias
Hi Tom, I initiated an issue here: https://issues.apache.org/jira/browse/SPARK-26121 Feel free to edit/update the ticket. If someone familiar with the codebase has any suggestion on the proper way of fixing this, I could work on it. Best, Anastasios On Mon, Nov 19, 2018 at 4:31 PM Tom Graves

Re: [Structured Streaming] Kafka group.id is fixed

2018-11-19 Thread Tom Graves
This makes sense to me and was going to propose something similar in order to be able to use the kafka acls more effectively as well, can you file a jira for it? Tom On Friday, November 9, 2018, 2:26:12 AM CST, Anastasios Zouzias wrote: Hi all, I run in the following situation with

Re: [Structured Streaming] Kafka group.id is fixed

2018-11-09 Thread Cody Koeninger
That sounds reasonable to me On Fri, Nov 9, 2018 at 2:26 AM Anastasios Zouzias wrote: > > Hi all, > > I run in the following situation with Spark Structure Streaming (SS) using > Kafka. > > In a project that I work on, there is already a secured Kafka setup where ops > can issue an SSL

[Structured Streaming] Kafka group.id is fixed

2018-11-09 Thread Anastasios Zouzias
Hi all, I run in the following situation with Spark Structure Streaming (SS) using Kafka. In a project that I work on, there is already a secured Kafka setup where ops can issue an SSL certificate per "group.id", which should be predefined (or hopefully its prefix to be predefined). On the