[GitHub] spark issue #23103: [SPARK-26121] [Structured Streaming] Allow users to defi...

2018-11-22 Thread zouzias 
Github user zouzias commented on the issue:

https://github.com/apache/spark/pull/23103
  
@koeninger, I will make the doc changes asap. FYI, I plan to make changes 
on file `structured-streaming-kafka-integration.md` seems the most relevant doc 
for this diff.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23103: [SPARK-26121] [Structured Streaming] Allow users ...

2018-11-21 Thread zouzias 
Github user zouzias commented on a diff in the pull request:

https://github.com/apache/spark/pull/23103#discussion_r235627877
  
--- Diff: 
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
 ---
@@ -538,6 +538,17 @@ private[kafka010] object KafkaSourceProvider extends 
Logging {
   .setIfUnset(ConsumerConfig.RECEIVE_BUFFER_CONFIG, 65536: 
java.lang.Integer)
   .build()
 
+  /**
+   * Returns a unique consumer group (group.id), allowing the user to set 
the prefix of
+   * the consumer group
+   */
+  private def streamingUniqueGroupId(parameters: Map[String, String],
+ metadataPath: String): String = {
+val groupIdPrefix = parameters
+  .getOrElse("group.id.prefix", "spark-kafka-source")
--- End diff --

Makes sense, fixed in 39424dd 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23103: [SPARK-26121] [Structured Streaming] Allow users ...

2018-11-21 Thread zouzias 
Github user zouzias commented on a diff in the pull request:

https://github.com/apache/spark/pull/23103#discussion_r235478266
  
--- Diff: 
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
 ---
@@ -538,6 +538,17 @@ private[kafka010] object KafkaSourceProvider extends 
Logging {
   .setIfUnset(ConsumerConfig.RECEIVE_BUFFER_CONFIG, 65536: 
java.lang.Integer)
   .build()
 
+  /**
+   * Returns a unique consumer group (group.id), allowing the user to set 
the prefix of
+   * the consumer group
+   */
+  private def streamingUniqueGroupId(parameters: Map[String, String],
--- End diff --

No worries, done.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23103: [SPARK-26121] [Structured Streaming] Allow users ...

2018-11-21 Thread zouzias 
Github user zouzias commented on a diff in the pull request:

https://github.com/apache/spark/pull/23103#discussion_r235478048
  
--- Diff: 
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
 ---
@@ -538,6 +538,17 @@ private[kafka010] object KafkaSourceProvider extends 
Logging {
   .setIfUnset(ConsumerConfig.RECEIVE_BUFFER_CONFIG, 65536: 
java.lang.Integer)
   .build()
 
+  /**
+   * Returns a unique consumer group (group.id), allowing the user to set 
the prefix of
+   * the consumer group
+   */
+  private def streamingUniqueGroupId(parameters: Map[String, String],
+ metadataPath: String): String = {
+val groupIdPrefix = parameters
+  .getOrElse("group.id.prefix", "spark-kafka-source")
--- End diff --

I named the property `kafka.groupIdPrefix` for now. Let me know if you a 
better name suggestion.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #23103: [SPARK-26121] [Structured Streaming] Allow users to defi...

2018-11-21 Thread zouzias 
Github user zouzias commented on the issue:

https://github.com/apache/spark/pull/23103
  
cc @koeninger 

Preliminary PR, todos:

* Decide on input parameter name for prefix consumer group, i.e., 
`streaming.group_id.prefix`.
* Document the above parameter in Spark SS docs.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23103: [SPARK-26121] init commit

2018-11-21 Thread zouzias 
GitHub user zouzias opened a pull request:

https://github.com/apache/spark/pull/23103

[SPARK-26121] init commit

## What changes were proposed in this pull request?

Allow the Spark Structured Streaming user to specify the prefix of the 
consumer group (group.id), compared to force consumer group ids of the form 
`spark-kafka-source-*`

## How was this patch tested?

Unit tests provided by Spark (backwards compatible change, i.e., user can 
optionally use the functionality)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Sqooba/spark SPARK-26121

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/23103.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #23103


commit 4afd3a4dd0794348be1462b896f20de4e344e5ce
Author: Anastasios Zouzias 
Date:   2018-11-21T10:24:53Z

[SPARK-26121] init commit




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org