[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/6862#issuecomment-113184801 I'm not a python programmer, but isn't the direct translation of that kafkaStreams = map(lambda _:KafkaUtils.createStream(...), range(0, numStreams)) Maybe append is more idiomatic... at any rate what's there looks like it will work --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6862#discussion_r32744362
--- Diff: docs/streaming-programming-guide.md ---
@@ -1937,6 +1937,16 @@ JavaPairDStreamString, String unifiedStream =
streamingContext.union(kafkaStre
unifiedStream.print();
{% endhighlight %}
/div
+div data-lang=python markdown=1
+{% highlight python %}
+numStreams = 5
+kafkaStreams = []
+for _ in range (numStreams):
+ kafkaStreams.append(KafkaUtils.createStream(...))
--- End diff --
Nit: List comprehension is more Pythonic
```
kafkaStreams = [KafkaUtils.createStream(...) for _ in range (numStreams)]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6862#issuecomment-113213947 [Test build #918 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/918/consoleFull) for PR 6862 at commit [`4bfd126`](https://github.com/apache/spark/commit/4bfd12617faef247e961ffbf94ebbafcb0930051). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user nssalian commented on a diff in the pull request:
https://github.com/apache/spark/pull/6862#discussion_r32748615
--- Diff: docs/streaming-programming-guide.md ---
@@ -1937,6 +1937,16 @@ JavaPairDStreamString, String unifiedStream =
streamingContext.union(kafkaStre
unifiedStream.print();
{% endhighlight %}
/div
+div data-lang=python markdown=1
+{% highlight python %}
+numStreams = 5
+kafkaStreams = []
+for _ in range (numStreams):
+ kafkaStreams.append(KafkaUtils.createStream(...))
--- End diff --
Made the changes as per @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6862#issuecomment-113213733 LGTM, merging into master, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6862 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user nssalian commented on the pull request: https://github.com/apache/spark/pull/6862#issuecomment-113211610 @davies and @koeninger Thank you for the comments. Do you think any other changes need to go in? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6862#issuecomment-113252231 [Test build #918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/918/console) for PR 6862 at commit [`4bfd126`](https://github.com/apache/spark/commit/4bfd12617faef247e961ffbf94ebbafcb0930051). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class IsNull(child: Expression) extends UnaryExpression with Predicate ` * `case class IsNotNull(child: Expression) extends UnaryExpression with Predicate ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/6862#issuecomment-113059639 Looks better, but in the end I don't know enough Python to be certain that's correct and canonical. @tdas @davies @koeninger do you have any thoughts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6862#discussion_r32658252
--- Diff: docs/streaming-programming-guide.md ---
@@ -1937,6 +1937,16 @@ JavaPairDStreamString, String unifiedStream =
streamingContext.union(kafkaStre
unifiedStream.print();
{% endhighlight %}
/div
+div data-lang=python markdown=1
+{% highlight python %}
+numStreams = 5
+kafkaStreams = []
+for x in range (0, numStreams):
+ kafkaStreams = x.map{ KafkaUtils.createStream(â¦)}
--- End diff --
Hm, I don't think this can be correct? x is an integer; you can't map it.
Python collections don't map like that anyway. Are you just trying to `append`
the new stream to `kafkaStreams`?
Nit: use `...` instead of the ellipsis character.
Is the method really `show()` in Pyspark instead of `print()`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8320] [Streaming] Add example in stream...
Github user nssalian commented on the pull request: https://github.com/apache/spark/pull/6862#issuecomment-112947266 @srowen , I changed the Kafka append, the loop structure and the print method call. Thank you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
