Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2556#issuecomment-57407523
I've merged this. Thanks for the fix!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2556
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2556#issuecomment-57058156
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/169/consoleFull)
for PR 2556 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2556#issuecomment-57060226
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/169/consoleFull)
for PR 2556 at commit
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18098324
--- Diff: docs/configuration.md ---
@@ -207,6 +207,25 @@ Apart from these, the following properties are also
available, and may be useful
/td
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2351
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56988133
Thanks for review this, your comments made it much better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-57024838
Whoops, looks like this failed unit tests and caused a build-break. I'm
going to revert it to un-break the build while we investigate.
---
If your project is set up
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/2556
[SPARK-3478] [PySpark] Profile the Python tasks
This patch add profiling support for PySpark, it will show the profiling
results
before the driver exits, here is one example:
```
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2556#issuecomment-57042267
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56859664
This looks good to me. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56863846
I noticed that we don't have any automated tests for `show_profiles()`, so
I tested it manually and found a problem when running this file through
`spark-submit`:
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56889367
@JoshRosen sorry for this mistake, fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56890919
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56891432
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/156/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56897188
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/156/consoleFull)
for PR 2351 at commit
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18067340
--- Diff: docs/configuration.md ---
@@ -207,6 +207,25 @@ Apart from these, the following properties are also
available, and may be useful
/td
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18067596
--- Diff: docs/configuration.md ---
@@ -207,6 +207,25 @@ Apart from these, the following properties are also
available, and may be useful
/td
/tr
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18070270
--- Diff: docs/configuration.md ---
@@ -207,6 +207,25 @@ Apart from these, the following properties are also
available, and may be useful
/td
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18071567
--- Diff: docs/configuration.md ---
@@ -207,6 +207,25 @@ Apart from these, the following properties are also
available, and may be useful
/td
/tr
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r17992850
--- Diff: python/pyspark/rdd.py ---
@@ -2081,8 +2085,44 @@ def _jrdd(self):
self.ctx.pythonExec,
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r17992966
--- Diff: python/pyspark/rdd.py ---
@@ -2081,8 +2085,44 @@ def _jrdd(self):
self.ctx.pythonExec,
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r17993579
--- Diff: docs/configuration.md ---
@@ -207,6 +207,22 @@ Apart from these, the following properties are also
available, and may be useful
/td
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r17993687
--- Diff: docs/configuration.md ---
@@ -207,6 +207,22 @@ Apart from these, the following properties are also
available, and may be useful
/td
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56723672
I really like this approach to profiling; it's a very clever use of
accumulators. My only feedback concerns UX / UI issues (see a few comments
above RE: configuration
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r17993979
--- Diff: python/pyspark/rdd.py ---
@@ -2081,8 +2085,44 @@ def _jrdd(self):
self.ctx.pythonExec,
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18002615
--- Diff: python/pyspark/rdd.py ---
@@ -2081,8 +2085,44 @@ def _jrdd(self):
self.ctx.pythonExec,
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18005654
--- Diff: python/pyspark/rdd.py ---
@@ -2081,8 +2085,44 @@ def _jrdd(self):
self.ctx.pythonExec,
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18005737
--- Diff: python/pyspark/rdd.py ---
@@ -2081,8 +2085,44 @@ def _jrdd(self):
self.ctx.pythonExec,
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56752778
@JoshRosen I had addressed your comments, plz take another look, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56752977
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20767/consoleFull)
for PR 2351 at commit
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18006453
--- Diff: python/pyspark/rdd.py ---
@@ -2025,6 +2025,7 @@ class PipelinedRDD(RDD):
rdd.flatMap(lambda x: [x, x]).reduce(add)
20
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18006482
--- Diff: python/pyspark/context.py ---
@@ -793,6 +796,40 @@ def runJob(self, rdd, partitionFunc, partitions=None,
allowLocal=False):
it =
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r18006502
--- Diff: docs/configuration.md ---
@@ -207,6 +207,25 @@ Apart from these, the following properties are also
available, and may be useful
/td
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56753544
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20767/
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56753598
(I killed the test here so that I could re-run it with the newer commits).
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56753750
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/147/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56753858
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20769/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56758654
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/147/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56758830
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20769/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56758832
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20769/
Github user shaneknapp commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56758906
jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56759309
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20771/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56763368
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20771/consoleFull)
for PR 2351 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56763374
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56599152
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20730/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56605488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20730/
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-56605482
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20730/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55928227
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20469/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55938151
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20469/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55483390
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20252/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55516221
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20297/consoleFull)
for PR 2351 at commit
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2351#discussion_r17509328
--- Diff: python/pyspark/accumulators.py ---
@@ -215,6 +215,21 @@ def addInPlace(self, value1, value2):
COMPLEX_ACCUMULATOR_PARAM =
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55480246
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20250/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55481451
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20250/consoleFull)
for PR 2351 at commit
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55482196
@JoshRosen I had addressed your comment, also added docs for configs and
tests.
I realized that the profile result also can be showed interactively, by
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55482240
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20252/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55482315
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20253/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55482333
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20253/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55224983
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/66/consoleFull)
for PR 2351 at commit
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/2351
[SPARK-3478] [PySpark] Profile the Python tasks
This patch add profiling support for PySpark, it will show the profiling
results
before the driver exits, here is one example:
```
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55207154
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/48/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55209533
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20128/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55211000
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/48/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55212763
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20128/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55216348
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20135/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55219072
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20135/consoleFull)
for PR 2351 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2351#issuecomment-55220913
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/66/consoleFull)
for PR 2351 at commit
68 matches
Mail list logo
