Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/11255#issuecomment-187830413
@srowen Yes, that was my intent. I'm not sure if there is a way to stop the
Scala shell from starting.
---
If your project is set up for it, you can reply to
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/11255#issuecomment-185936031
retest please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
GitHub user djalova opened a pull request:
https://github.com/apache/spark/pull/11255
[SPARK-12759][Core][Spark should fail fast if --executor-memory is too
small for spark to start]
Added an exception to be thrown in UnifiedMemoryManager.scala if the
configuration given for
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156591380
Sorry I wasn't reading your comment carefully. I'll make the update.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156588293
@zsxwing The test right now doesn't even return any streamInputInfos or
outputOperationInfos. In the Scala tests for the StreamingListener, a custom
Receiver was
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-156523410
Sure, I'll change it to check for at least 1 batch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as wel
Github user djalova commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r44694463
--- Diff: python/pyspark/streaming/tests.py ---
@@ -403,6 +404,80 @@ def func(dstream):
self._test_func(input, func, expected
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155923601
There also doesn't seem to be a way to add receivers using Python. I've
been testing the code on my own machine using socketTextStream() but I don'
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155908396
> Moreover, it would be better if you can access the contents of Map fields
using Python dict syntax.
Do you have any suggestions for doing this? I was us
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155648728
I see, that worked. Should I keep the Python classes for the Infos or just
have the user make Java method calls to get the values?
---
If your project is set up for it
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155615807
> Could you create a PythonStreamingListener interface in Java side?
I could do that but I would either have to have it extend StreamingListener
which could be a
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155593098
I'm unsure what you mean. In the case that a user wants to override one of
the StreamingListener methods in Python, don't we want to make sure that the
pyt
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155583518
@zsxwing Since JavaStreamingListener is an actual class and not an
interface in the jvm, I can't use "implements" in py4j. How should I work
around this
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-155573413
I'll work on it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9420#issuecomment-154488931
Will this be merged soon?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9420#issuecomment-153909707
Can you test this again please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-153162764
@zsxwing Can you take a look at my changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-151615522
I added in your suggestions. Let me know if we should still implement a
Java API for it.
---
If your project is set up for it, you can reply to this email and have
Github user djalova commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r42926305
--- Diff: python/pyspark/streaming/tests.py ---
@@ -398,6 +399,69 @@ def func(dstream):
self._test_func(input, func, expected
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-150713407
The listener sometimes receives 3 batchCompleted events instead of 4 when I
run the test. However, when I add a minor delay the test passes consistently.
Can I get
Github user djalova commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r42782220
--- Diff: python/pyspark/streaming/listener.py ---
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Github user djalova commented on a diff in the pull request:
https://github.com/apache/spark/pull/9186#discussion_r4248
--- Diff: python/pyspark/streaming/listener.py ---
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-149991651
@tdas it looks like Jenkins didn't retest
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user djalova commented on the pull request:
https://github.com/apache/spark/pull/9186#issuecomment-149739513
The test that failed was a flaky StreamingKMeansTest.
Please retest.
---
If your project is set up for it, you can reply to this email and have your
reply appear
GitHub user djalova opened a pull request:
https://github.com/apache/spark/pull/9186
[SPARK-6328] [pyspark]
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/djalova/spark SPARK-6328
Alternatively you can review and apply these
25 matches
Mail list logo
