[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 Many thanks, Xiao. I learnt lots. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 @merlintang Can you close this PR? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 Thanks! Merging to 1.6 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70990/ Test PASSed. ---

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70990 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70990/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 LGTM pending test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70990/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70964 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70964/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70964/ Test FAILed. ---

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70964/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 Weird... Not sure why the build failed. The build works in my local environment. cc @srowen @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70908/ Test FAILed. ---

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70908/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70908/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70907/ Test FAILed. ---

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70907 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70907/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70907/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @gatorsmile can you retest the patch, then we can merge. Sorry to ping you multiple times since several users are asking this. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70785/ Test PASSed. ---

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70785 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70785/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70785 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70785/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @gatorsmile I have backport the test case in #16339 with small modification. because the "INSERT OVERWRITE TABLE tab SELECT '$i'" will bring the issue from hive side e.g.,

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 yes, let me backport the test cases for checking the staging file. On Thu, Dec 29, 2016 at 10:11 PM, Xiao Li wrote: > Is that possible to backport the

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 Is that possible to backport the test cases in https://github.com/apache/spark/pull/16399? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 Thanks, Wenchen, I have backport the code of #16339 to here, I have tested it locally. Can you review and verify? On Sun, Dec 25, 2016 at 11:04 PM, Wenchen Fan

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15819 https://github.com/apache/spark/pull/16399 has been merged, feel free if you wanna backport to 1.6 --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @gatorsmile Great! thanks so much, because I was pinged multiple times for this bug. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 Let me backport it to Spark 2.0.x at first. We need to follow the backport ordering --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @cloud-fan @gatorsmile I have backport the code from #16134, can you verify and backport this to spark 1.6.x? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @gatorsmile one more customer is running into this issue in the spark 1.6.x. I backport the code #16134 to here and test it manually. Please verify. --- If your project is set up for it, you

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @cloud-fan @gatorsmile this patch is related to #16134, It seems #16134 would be merged soon. Meanwhile, should we backport #16104 into 1.6.x? please advise. or else, I just backport #16134 and

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 Great, once the #16134 is done, we can backport them together. On Tue, Dec 13, 2016 at 12:18 AM, Wenchen Fan

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15819 yea, I think we should backport a complete staging dir cleanup functionality to 1.6, let's wait for #16134 --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 The current fix does not resolve the issue when users hitting abnormal termination of JVM. In addition, if the JVM does not stop, these temporary files could consume a lot of spaces. Thus, I

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @gatorsmile what is going on this patch? this is a backport code, thus, can you merge this patch into 1.6.x ? more than one users are running into this issue in the spark 1.6.x. --- If your

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 @lichenglin Another PR https://github.com/apache/spark/pull/16134 is trying to delete the staging directory and the temporary data files (which is pretty big in your case) after each insert.

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user lichenglin commented on the issue: https://github.com/apache/spark/pull/15819 In fact,I'm using zeppelin to run sql. When I restart spark interpreter , the folders are deleted. Thank you a lot --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 do you exit the spark shell ? I have tested on this, and this staging file would be removed after we exit the spark shell under spark 2.0.x. meanwhile, the staging file are used for

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user lichenglin commented on the issue: https://github.com/apache/spark/pull/15819 here is some result for `du -h --max-depth=1 .` 3.3G./.hive-staging_hive_2016-12-06_18-17-48_899_1400956608265117052-5 13G

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 Could you post the layout of that staging folder? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user lichenglin commented on the issue: https://github.com/apache/spark/pull/15819 I'm using spark 2.0.2 I got a really big hive-stage folder. May I delete the folder Manually ? does it make any bad affect on warehouse? --- If your project is set up for it, you can

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 @cloud-fan this is related to this PR in the 2.0.x https://github.com/apache/spark/pull/12770 --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15819 ok @merlintang can you find out which PR adds these codes to 2.0? Then other people can know what we are backporting in this PR --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 Ok. On Sun, Dec 4, 2016 at 6:25 PM, Reynold Xin wrote: > We have stopped making new releases for 1.5 so it makes no sense to > backport. >

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user rxin commented on the issue: https://github.com/apache/spark/pull/15819 We have stopped making new releases for 1.5 so it makes no sense to backport. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 this bug is related to 1.5.x as well as 1.6.x. please backport to 1.5.x as well. On Sun, Dec 4, 2016 at 6:20 PM, Reynold Xin wrote: > If it is

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user rxin commented on the issue: https://github.com/apache/spark/pull/15819 If it is a bug fix and low risk, sure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15819 OK so the problem becomes, do we want to backport this to 1.6? cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

Github user merlintang commented on the issue: https://github.com/apache/spark/pull/15819 it is updated. On Sun, Dec 4, 2016 at 11:23 AM, Xiao Li wrote: > @merlintang Could you please add > [Branch-1.6] in