[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-26 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17343 LGTM. Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77430/ Test PASSed. ---

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #77430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77430/testReport)** for PR 17343 at commit [`4bc6e3e`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-26 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17343 LGTM pending tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #77430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77430/testReport)** for PR 17343 at commit [`4bc6e3e`](https://github.com/apache/spark/commit/4b

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-25 Thread sameeragarwal
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/17343 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-25 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/17343 cc @zsxwing Could you find some time to review this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77382/ Test PASSed. ---

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #77382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77382/testReport)** for PR 17343 at commit [`d4f09c2`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-25 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17343 Thanks @sameeragarwal, addressed the check style issues. Yes, the exisiting unit tests in UnsafeShuffleWriter#mergeSpillsWithTransferToAndLZF covers this code. --- If your project is set up fo

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #77382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77382/testReport)** for PR 17343 at commit [`d4f09c2`](https://github.com/apache/spark/commit/d4

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-05-03 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17343 ping @zsxwing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-24 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17343 LGTM will wait a bit to allow for others to comment. @zsxwing can you also take a look ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-24 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17343 ping @rxin, @mridulm. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74897/ Test PASSed. ---

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74897/testReport)** for PR 17343 at commit [`06c1909`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74893/ Test PASSed. ---

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74893 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74893/testReport)** for PR 17343 at commit [`5fe279e`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74897/testReport)** for PR 17343 at commit [`06c1909`](https://github.com/apache/spark/commit/06

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17343 @rxin - Updated documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74893/testReport)** for PR 17343 at commit [`5fe279e`](https://github.com/apache/spark/commit/5f

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17343 Can you add some documentation inline so in the future we'd know why specific implementations were chosen? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-19 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17343 Ah, looks like I missed that CountingOutputStream was introduced after BOS and not before. Looks good to me. --- If your project is set up for it, you can reply to this email and have your repl

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-19 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17343 Background - you need to do a flush() to ensure the indices generated are valid. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If y

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-19 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17343 If we make flush() noop, then buffered (uncommitted) data wont be written to the stream; am I missing something here, or is this change broken ? --- If your project is set up for it, you can reply

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74805/ Test PASSed. ---

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74805/testReport)** for PR 17343 at commit [`368dd29`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74805/testReport)** for PR 17343 at commit [`368dd29`](https://github.com/apache/spark/commit/36

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74802/ Test PASSed. ---

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74802/testReport)** for PR 17343 at commit [`00da825`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74800/ Test FAILed. ---

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74800/testReport)** for PR 17343 at commit [`1834db6`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74798/ Test FAILed. ---

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17343 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74798/testReport)** for PR 17343 at commit [`e9ac76e`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74802/testReport)** for PR 17343 at commit [`00da825`](https://github.com/apache/spark/commit/00

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74800/testReport)** for PR 17343 at commit [`1834db6`](https://github.com/apache/spark/commit/18

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17343 cc - @rxin, @squito, @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled a

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17343 **[Test build #74798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74798/testReport)** for PR 17343 at commit [`e9ac76e`](https://github.com/apache/spark/commit/e9