[GitHub] spark pull request: SPARK-3177 (on Master Branch)

2014-09-03 Thread ScrapCodes
Github user ScrapCodes commented on a diff in the pull request: https://github.com/apache/spark/pull/2204#discussion_r17032605 --- Diff: yarn/common/src/test/scala/org/apache/spark/deploy/yarn/ClientBaseSuite.scala --- @@ -232,6 +233,15 @@ class ClientBaseSuite extends FunSuite

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2240#issuecomment-54255463 I talked to @shivaram offline. It is not really going to simplify review if I break this into two PRs because the renaming part is very small and has only about ~50 loc

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread colorant
Github user colorant commented on a diff in the pull request: https://github.com/apache/spark/pull/2240#discussion_r17032661 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -223,14 +226,14 @@ object SparkEnv extends Logging { val

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2240#issuecomment-54255610 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19639/consoleFull) for PR 2240 at commit

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2240#discussion_r17032690 --- Diff: core/src/main/scala/org/apache/spark/network/nio/NioBlockTransferService.scala --- @@ -0,0 +1,205 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-3309] [PySpark] Put all public API in _...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2205#issuecomment-54255622 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19640/consoleFull) for PR 2205 at commit

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2240#issuecomment-54255668 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19639/consoleFull) for PR 2240 at commit

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2240#discussion_r17032727 --- Diff: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala --- @@ -0,0 +1,270 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2240#discussion_r17032815 --- Diff: core/src/main/scala/org/apache/spark/network/BlockFetchingListener.scala --- @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/2240#discussion_r17032847 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -223,14 +226,14 @@ object SparkEnv extends Logging { val shuffleMemoryManager

[GitHub] spark pull request: [SPARK-3335] [SQL] [PySpark] support broadcast...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2243#issuecomment-54256256 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19630/consoleFull) for PR 2243 at commit

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/2231#discussion_r17033162 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/classification/LogisticRegressionSuite.scala --- @@ -125,4 +125,39 @@ class LogisticRegressionSuite

[GitHub] spark pull request: SPARK-1793 - Heavily duplicated test setup cod...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/726#issuecomment-54257541 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19635/consoleFull) for PR 726 at commit

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2231#issuecomment-54257879 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19633/consoleFull) for PR 2231 at commit

[GitHub] spark pull request: [SPARK-3310][SQL] Directly use currentTable wi...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2203#issuecomment-54258880 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19631/consoleFull) for PR 2203 at commit

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2240#issuecomment-54259036 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19641/consoleFull) for PR 2240 at commit

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54259070 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19642/consoleFull) for PR 2229 at commit

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2240#issuecomment-54259106 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19641/consoleFull) for PR 2240 at commit

[GitHub] spark pull request: [SPARK-3280] Made sort-based shuffle the defau...

2014-09-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2178#issuecomment-54259281 Ok submitted a PR to remove spark.shuffle.spill.compress: https://github.com/apache/spark/pull/2247 --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/2247 [SPARK-3367] Remove spark.shuffle.spill.compress Replaced it with existing spark.shuffle.compress. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: [SPARK-3309] [PySpark] Put all public API in _...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2205#issuecomment-54259512 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19640/consoleFull) for PR 2205 at commit

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...

2014-09-03 Thread baishuo
Github user baishuo commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-54259701 Hi @marmbrus and @liancheng, the latest code had pass dev/lint-scala and sbt/sbt catalyst/test sql/test hive/test locally. --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2240#issuecomment-54259809 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19644/consoleFull) for PR 2240 at commit

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54259798 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19643/consoleFull) for PR 2247 at commit

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...

2014-09-03 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2226#discussion_r17034303 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala --- @@ -186,32 +298,42 @@ case class InsertIntoHiveTable(

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...

2014-09-03 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-54260384 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...

2014-09-03 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-54260377 Please also submit golden answer files for newly whitelisted test cases in `HiveCompatibilitySuite`. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-2973][SQL] Lightweight SQL commands wit...

2014-09-03 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2215#issuecomment-54260952 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-2219][SQL] Added support for the add j...

2014-09-03 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2242#issuecomment-54260987 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-3362][SQL] bug in casewhen resolve

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2245#issuecomment-54261122 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19636/consoleFull) for PR 2245 at commit

[GitHub] spark pull request: [SPARK-3327] Make broadcasted value mutable fo...

2014-09-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2217#issuecomment-54261213 Why do you need to change the mutability of broadcast variable in order to enable that? In general, immutability makes the whole thing safer. You should still be

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread BigCrunsh
Github user BigCrunsh commented on a diff in the pull request: https://github.com/apache/spark/pull/2231#discussion_r17034656 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/classification/LogisticRegressionSuite.scala --- @@ -125,4 +125,39 @@ class LogisticRegressionSuite

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread BigCrunsh
Github user BigCrunsh commented on the pull request: https://github.com/apache/spark/pull/2231#issuecomment-54261292 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2219][SQL] Added support for the add j...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2242#issuecomment-54261261 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19645/consoleFull) for PR 2242 at commit

[GitHub] spark pull request: [SPARK-3362][SQL] bug in casewhen resolve

2014-09-03 Thread adrian-wang
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2245#issuecomment-54261288 The failure is in PageRankSuite, test it again, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-2973][SQL] Lightweight SQL commands wit...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2215#issuecomment-54261286 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19646/consoleFull) for PR 2215 at commit

[GitHub] spark pull request: [SPARK-3362][SQL] bug in casewhen resolve

2014-09-03 Thread adrian-wang
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2245#issuecomment-54261370 and the first failure is in pyspark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-03 Thread adrian-wang
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2246#issuecomment-54261566 JsonRDD also called HiveTypeCoercion.allPromotion, but with a method owned by itself, which lead to the failure, I have fix that also. --- If your project is set up

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2231#issuecomment-54261656 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19647/consoleFull) for PR 2231 at commit

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2246#issuecomment-54262054 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19648/consoleFull) for PR 2246 at commit

[GitHub] spark pull request: [SQL] Refined Thrift server test suite

2014-09-03 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2214#issuecomment-54262123 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SQL] Refined Thrift server test suite

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2214#issuecomment-54262408 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19649/consoleFull) for PR 2214 at commit

[GitHub] spark pull request: [SPARK-2961][SQL] Use statistics to prune batc...

2014-09-03 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2188#issuecomment-54262405 @marmbrus Addressed all the comments, thanks for the detailed review! --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-2961][SQL] Use statistics to prune batc...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2188#issuecomment-54262762 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19650/consoleFull) for PR 2188 at commit

[GitHub] spark pull request: [SQL] Refined Thrift server test suite

2014-09-03 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2214#issuecomment-54263650 The test failures only occur when running on Jenkins, I couldn't reproduce it either locally or on Jenkins server node. May fire some debugging commits later to

[GitHub] spark pull request: [SPARK-3019] Pluggable block transfer interfac...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2240#issuecomment-54264464 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19644/consoleFull) for PR 2240 at commit

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54264487 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19643/consoleFull) for PR 2247 at commit

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54264700 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54265330 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19651/consoleFull) for PR 2247 at commit

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54266318 Is this being done to workaround the bug (sort shuffle breakage) ? The way we use it is, not use compress but only enable it if spilled to disk (a corner case) -

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54266574 What compression overhead are you talking about? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54266944 The overhead of compressing ? On 03-Sep-2014 2:02 pm, Reynold Xin notificati...@github.com wrote: What compression overhead are you talking about?

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54267261 I don't get it. Are you saying compression is turned on for spill files, but not for general shuffle files? And in response to your original question about sort

[GitHub] spark pull request: [SPARK-3273][SPARK-3302][SPARK-3301]We should ...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2175#issuecomment-54267792 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19652/consoleFull) for PR 2175 at commit

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54267869 Yes, exactly : with higher block size configured sometimes. Most tasks will not incur (de)compression overhead except for the very few which spill due to skew.

[GitHub] spark pull request: [SPARK-3301]The spark version in the welcome m...

2014-09-03 Thread witgo
Github user witgo closed the pull request at: https://github.com/apache/spark/pull/2196 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-3301]The spark version in the welcome m...

2014-09-03 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2196#issuecomment-54267918 @ScrapCodes @nchammas I merge this PR into #2175 and close this PR --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: SPARK-1719: spark.*.extraLibraryPath isn't app...

2014-09-03 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1022#issuecomment-54268060 I merge this PR into #1031 and close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-1719: spark.*.extraLibraryPath isn't app...

2014-09-03 Thread witgo
Github user witgo closed the pull request at: https://github.com/apache/spark/pull/1022 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-3280] Made sort-based shuffle the defau...

2014-09-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/2178#issuecomment-54268120 I think this is working fine for hash based shuffle ? So is it a bug that needs to be fixed as opposed to worked around. --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54268199 Do you have any data on a job in which disabling shuffle compression helps with performance substantially (especially with sort-based shuffle in which the number of

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2231#issuecomment-54268559 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19647/consoleFull) for PR 2231 at commit

[GitHub] spark pull request: [SPARK-2096][SQL] Correctly parse dot notation...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2230#issuecomment-54268698 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19653/consoleFull) for PR 2230 at commit

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54269185 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19642/consoleFull)** after a configured wait of `120m`. --- If your project

[GitHub] spark pull request: [SPARK-2096][SQL] Correctly parse dot notation...

2014-09-03 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/2230#issuecomment-54269552 @marmbrus Sorry for missing the `distinct`. Since we parse the dot in `SqlParser` now, the only possible formats of `name` passed into `LogicalPlan.resolve` is ident

[GitHub] spark pull request: [SPARK-2961][SQL] Use statistics to prune batc...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2188#issuecomment-54269670 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19650/consoleFull) for PR 2188 at commit

[GitHub] spark pull request: [SPARK-2973][SQL] Lightweight SQL commands wit...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2215#issuecomment-54270387 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19646/consoleFull) for PR 2215 at commit

[GitHub] spark pull request: [SPARK-2219][SQL] Added support for the add j...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2242#issuecomment-54270765 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19645/consoleFull) for PR 2242 at commit

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54271426 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19651/consoleFull) for PR 2247 at commit

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread BigCrunsh
Github user BigCrunsh commented on the pull request: https://github.com/apache/spark/pull/2231#issuecomment-54271799 I am not sure why this is failing... ``` - local metrics *** FAILED *** [info] org.apache.spark.scheduler.StageInfo@2f3ab430 fetchWaitTime

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread BigCrunsh
Github user BigCrunsh commented on the pull request: https://github.com/apache/spark/pull/2231#issuecomment-54271813 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2246#issuecomment-54271963 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19648/consoleFull) for PR 2246 at commit

[GitHub] spark pull request: [SQL] Refined Thrift server test suite

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2214#issuecomment-54271972 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19649/consoleFull) for PR 2214 at commit

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2231#issuecomment-54272192 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19654/consoleFull) for PR 2231 at commit

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-03 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54272480 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...

2014-09-03 Thread mridulm
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54272557 I think we are looking at it the wrong way - this is an exposed interface which is being removed in a minor release. It might also break direct usage of

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54273177 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19655/consoleFull) for PR 2229 at commit

[GitHub] spark pull request: [SPARK-1720] Add the value of LD_LIBRARY_PATH ...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-54273176 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19656/consoleFull) for PR 1031 at commit

[GitHub] spark pull request: [SPARK-3273][SPARK-3302][SPARK-3301]We should ...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2175#issuecomment-54273894 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19652/consoleFull) for PR 2175 at commit

[GitHub] spark pull request: [SPARK-3372] [MLLIB] Scalastyle fails due to U...

2014-09-03 Thread sarutak
GitHub user sarutak opened a pull request: https://github.com/apache/spark/pull/2248 [SPARK-3372] [MLLIB] Scalastyle fails due to UTF-8 character being contained in Gradient.scala You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: [SPARK-3372] [MLLIB] Scalastyle fails due to U...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2248#issuecomment-54274677 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19657/consoleFull) for PR 2248 at commit

[GitHub] spark pull request: SPARK-3211 .take() is OOM-prone with empty par...

2014-09-03 Thread ash211
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/2117#issuecomment-54276862 Fixed the same in Python, corrected the comment, and switched from *2 to *4 for faster take() on mostly-empty partition. Let's see how the unit tests go. --- If

[GitHub] spark pull request: SPARK-3211 .take() is OOM-prone with empty par...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2117#issuecomment-54277144 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19658/consoleFull) for PR 2117 at commit

[GitHub] spark pull request: [SPARK-3327] Make broadcasted value mutable fo...

2014-09-03 Thread viirya
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/2217#issuecomment-54278826 @rxin. I need a way to modify broadcasted variables locally and keep those variables for later use. The locally modified variables are used to store some values

[GitHub] spark pull request: [SPARK-1720] Add the value of LD_LIBRARY_PATH ...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1031#issuecomment-54278998 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19656/consoleFull) for PR 1031 at commit

[GitHub] spark pull request: [SPARK-2096][SQL] Correctly parse dot notation...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2230#issuecomment-54279166 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19653/consoleFull) for PR 2230 at commit

[GitHub] spark pull request: [SPARK-3327] Make broadcasted value mutable fo...

2014-09-03 Thread viirya
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/2217#issuecomment-54279202 @rxin, I can get the idea that immutability makes the whole thing safer for broadcasted variables. So I am just wondering if it is worth providing such mutability for the

[GitHub] spark pull request: [FIX][MLLIB] Use SquaredL2Updater in LogisticR...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2231#issuecomment-54279965 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19654/consoleFull) for PR 2231 at commit

[GitHub] spark pull request: [SPARK-3372] [MLLIB] Scalastyle fails due to U...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2248#issuecomment-54280386 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19657/consoleFull) for PR 2248 at commit

[GitHub] spark pull request: [SPARK-3327] Make broadcasted value mutable fo...

2014-09-03 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2217#issuecomment-54281202 @viirya I also don't quite get why mutability helps the use case you describe. You seem to want some persistent mutable state per executor. But that isn't the purpose of

[GitHub] spark pull request: SPARK-3211 .take() is OOM-prone with empty par...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2117#issuecomment-54282260 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19658/consoleFull) for PR 2117 at commit

[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...

2014-09-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54283316 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19655/consoleFull) for PR 2229 at commit

[GitHub] spark pull request: [SPARK-2240][SQL]Spark SQL add LeftSemiBloomFi...

2014-09-03 Thread YanjieGao
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1127#issuecomment-54283611 Hi marmbrus , Got it , if i have some other good idea i will try to communicate with you ,Thanks ,I will close it latter. --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-2240][SQL]Spark SQL add LeftSemiBloomFi...

2014-09-03 Thread YanjieGao
Github user YanjieGao closed the pull request at: https://github.com/apache/spark/pull/1127 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2236][SQL]SparkSQL add SkewJoin

2014-09-03 Thread YanjieGao
Github user YanjieGao closed the pull request at: https://github.com/apache/spark/pull/1134 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2236][SQL]SparkSQL add SkewJoin

2014-09-03 Thread YanjieGao
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1134#issuecomment-54283740 Hi marmbrus,I will close it. Best Regards --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2237][CORE]Add ZLIBCompressionCodec cod...

2014-09-03 Thread YanjieGao
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1121#issuecomment-54283956 Ok,i thinks so ,may be some small third-party lib may cause uncertain problem , I will close this PR. If have some mature solution , i will send PR and make a

[GitHub] spark pull request: [SPARK-2237][CORE]Add ZLIBCompressionCodec cod...

2014-09-03 Thread YanjieGao
Github user YanjieGao closed the pull request at: https://github.com/apache/spark/pull/1121 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: SPARK-3211 .take() is OOM-prone with empty par...

2014-09-03 Thread ash211
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/2117#issuecomment-54284918 Failed unit test seems unrelated: ``` - Count two triangles java.io.IOException: PARSING_ERROR(2)

[GitHub] spark pull request: [SPARK-3187] [yarn] Cleanup allocator code.

2014-09-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2169 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

  1   2   3   4   5   >