Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-157142357
Thanks @djalova and @zsxwing for your efforts. Merging this to master and
1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9186
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156361543
#9669 has been merged. please merge with master and test the PR again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156499802
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156522465
@djalova could you fix the test?
```
[Running ]
test_batch_info_reports (__main__.StreamingListenerTests) ... FAIL
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156515638
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156515644
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156515420
**[Test build #45874 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45874/consoleFull)**
for PR 9186 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156532648
**[Test build #45880 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45880/consoleFull)**
for PR 9186 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156501246
**[Test build #45874 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45874/consoleFull)**
for PR 9186 at commit
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156523410
Sure, I'll change it to check for at least 1 batch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156546057
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156545862
**[Test build #45880 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45880/consoleFull)**
for PR 9186 at commit
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156579740
@djalova could you add the following checks?
> I just meant you can access the Map here, e.g.:
>
> for streamId in info.streamIdToInputInfo():
>
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156586652
**[Test build #45905 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45905/consoleFull)**
for PR 9186 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156546059
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156585296
Sorry for my unclear comment. I just meant adding the real codes to access
fields of `streamInputInfo` and `outputOperationInfo`, such as
```
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156595996
**[Test build #45906 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45906/consoleFull)**
for PR 9186 at commit
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156625409
Thanks @djalova LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156602905
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156602904
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156602806
**[Test build #45906 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45906/consoleFull)**
for PR 9186 at commit
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156588293
@zsxwing The test right now doesn't even return any streamInputInfos or
outputOperationInfos. In the Scala tests for the StreamingListener, a custom
Receiver was used
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156591380
Sorry I wasn't reading your comment carefully. I'll make the update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156593894
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156593893
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156593862
**[Test build #45905 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45905/consoleFull)**
for PR 9186 at commit
Github user djalova commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r44694463
--- Diff: python/pyspark/streaming/tests.py ---
@@ -403,6 +404,80 @@ def func(dstream):
self._test_func(input, func, expected)
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156198315
We should retest this PR after merging #9669.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155710041
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155710043
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r44563303
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingListener.scala
---
@@ -18,6 +18,82 @@
package
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155863868
Could you add some codes in tests to demonstrate all fields in
`JavaBatchInfo`, `JavaReceiverInfo` and `JavaOutputOperationInfo` can be
accessed in Python? It's not
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155864069
BTW, LGTM for the new Python APIs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155709852
**[Test build #45611 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45611/consoleFull)**
for PR 9186 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-15598
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155966552
**[Test build #45690 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45690/consoleFull)**
for PR 9186 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-15599
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r44626024
--- Diff: python/pyspark/streaming/tests.py ---
@@ -403,6 +404,80 @@ def func(dstream):
self._test_func(input, func, expected)
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r44626086
--- Diff: python/pyspark/streaming/tests.py ---
@@ -403,6 +404,80 @@ def func(dstream):
self._test_func(input, func, expected)
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r44626089
--- Diff: python/pyspark/streaming/tests.py ---
@@ -403,6 +404,80 @@ def func(dstream):
self._test_func(input, func, expected)
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r44626076
--- Diff: python/pyspark/streaming/tests.py ---
@@ -403,6 +404,80 @@ def func(dstream):
self._test_func(input, func, expected)
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156017667
I just noticed that Streaming Python unit tests cannot report failure. It
always says `pass` even if some test fails. Investigating it.
---
If your project is set up
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155954947
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155954931
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155955139
**[Test build #45690 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45690/consoleFull)**
for PR 9186 at commit
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155889295
Just a point, if you can add this very soon, then I can merge it for 1.6
release.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155908396
> Moreover, it would be better if you can access the contents of Map fields
using Python dict syntax.
Do you have any suggestions for doing this? I was using
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155923601
There also doesn't seem to be a way to add receivers using Python. I've
been testing the code on my own machine using socketTextStream() but I don't
think that's
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155941553
Yeah, that is hard. We can figure that out in a later PR. Can you atleast
test that the contents of the maps in the BatchInfo can be accesses in
python? Should be
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155572841
@djalova could you update this PR soon? This is very helpful for 1.6. If
you don't have time now, I can take it over. The commit to master will still
show up as yours.
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155573413
I'll work on it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155583518
@zsxwing Since JavaStreamingListener is an actual class and not an
interface in the jvm, I can't use "implements" in py4j. How should I work
around this?
---
If your
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155586759
> @zsxwing Since JavaStreamingListener is an actual class and not an
interface, I can't use "implements" in py4j. How should I work around this?
Should I change
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155593098
I'm unsure what you mean. In the case that a user wants to override one of
the StreamingListener methods in Python, don't we want to make sure that the
python method
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155606846
> I'm unsure what you mean. In the case that a user wants to override one
of the StreamingListener methods in Python, don't we want to make sure that the
python method
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155625169
I image adding these two classes:
```
trait PythonStreamingListener {
...
}
class PythonStreamingListenerWrapper(listener:
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155681206
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155681209
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155662537
> I see, that worked. Should I keep the Python classes for the Infos or
just have the user make Java method calls to get the values?
Just let the user use Java
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155673149
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155673160
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155673623
**[Test build #45599 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45599/consoleFull)**
for PR 9186 at commit
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155648728
I see, that worked. Should I keep the Python classes for the Infos or just
have the user make Java method calls to get the values?
---
If your project is set up for
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155255131
#9420 has been merged. could you please update your PR using the
JavaStreamingListener?
---
If your project is set up for it, you can reply to this email and have your
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-153162764
@zsxwing Can you take a look at my changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-153166260
I just sent #9420 to add a Java version of StreamingListener. It will make
this PR much easier.
---
If your project is set up for it, you can reply to this email and
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r43113846
--- Diff: python/pyspark/streaming/tests.py ---
@@ -398,6 +399,69 @@ def func(dstream):
self._test_func(input, func, expected)
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151473247
I'm thinking maybe we should add a Java API for StreamingListener at first,
it would make the PR simpler since most of Java collections can be mapped to
Python classes
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151626302
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151626307
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151616399
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151616357
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151619029
**[Test build #44459 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44459/consoleFull)**
for PR 9186 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151626049
**[Test build #44459 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44459/consoleFull)**
for PR 9186 at commit
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151615522
I added in your suggestions. Let me know if we should still implement a
Java API for it.
---
If your project is set up for it, you can reply to this email and have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151298325
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151298181
**[Test build #44377 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44377/consoleFull)**
for PR 9186 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151298326
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151293587
**[Test build #44377 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44377/consoleFull)**
for PR 9186 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151291830
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151291848
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r43078501
--- Diff: python/pyspark/streaming/listener.py ---
@@ -0,0 +1,285 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r43078141
--- Diff: python/pyspark/streaming/listener.py ---
@@ -0,0 +1,285 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r43078607
--- Diff: python/pyspark/streaming/listener.py ---
@@ -0,0 +1,285 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150663329
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150663302
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150665420
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150665416
**[Test build #44248 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44248/consoleFull)**
for PR 9186 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150665299
**[Test build #44248 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44248/consoleFull)**
for PR 9186 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150665419
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user djalova commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r42926305
--- Diff: python/pyspark/streaming/tests.py ---
@@ -398,6 +399,69 @@ def func(dstream):
self._test_func(input, func, expected)
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150713407
The listener sometimes receives 3 batchCompleted events instead of 4 when I
run the test. However, when I add a minor delay the test passes consistently.
Can I get
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150713359
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150713342
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150714210
**[Test build #44268 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44268/consoleFull)**
for PR 9186 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150717684
**[Test build #44268 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44268/consoleFull)**
for PR 9186 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150717751
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150717749
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r42715990
--- Diff: python/pyspark/streaming/listener.py ---
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
1 - 100 of 121 matches
Mail list logo
