[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2018-10-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2018-07-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93051/ Test FAILed. ---

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2018-07-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2018-07-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #93051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93051/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2018-07-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #93051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93051/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2018-07-15 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19955 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2018-06-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2018-01-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2018-01-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84983/ Test FAILed. ---

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #84983 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84983/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #84983 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84983/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84971/ Test FAILed. ---

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #84971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84971/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #84971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84971/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread ericvandenbergfb
Github user ericvandenbergfb commented on the issue: https://github.com/apache/spark/pull/19955 @sitalkedia Added comment in test section on some benchmarks. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread ericvandenbergfb
Github user ericvandenbergfb commented on the issue: https://github.com/apache/spark/pull/19955 @cloud-fan In this case it reverts to the existing behavior, it will spill synchronously, free memory and then continue inserting. In addition, it would re-tune (down) the spill

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-15 Thread ericvandenbergfb
Github user ericvandenbergfb commented on the issue: https://github.com/apache/spark/pull/19955 @cloud-fan I think this could be enabled for other sorters (mentioned that at end of design spec), just wanted to get this in incrementally. ---

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-13 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/19955 @ericvandenbergfb - In the testing section, let's put the benchmark numbers for large shuffle heavy jobs we ran internally. ---

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-13 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19955 and also what's the behavior if the memory is used up but `spark.shuffle.async.num.sorter` is not hit yet? --- - To

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-13 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19955 shall we do it for all the external sorters? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-13 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19955 cc @JoshRosen @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84889/ Test FAILed. ---

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #84889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84889/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #84889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84889/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84800/ Test FAILed. ---

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #84800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84800/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-12 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19955 **[Test build #84800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84800/testReport)** for PR 19955 at commit

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-12 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19955 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19955: [SPARK-21867][CORE] Support async spilling in UnsafeShuf...

2017-12-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19955 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional