[GitHub] spark pull request: SPARK-2553. CoGroupedRDD unnecessarily allocat...

2014-07-17 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1461#issuecomment-49401094 Merged this, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fea

[GitHub] spark pull request: SPARK-2553. CoGroupedRDD unnecessarily allocat...

2014-07-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1461 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [MLlib] SPARK-1536: multiclass classification ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/886#issuecomment-49401055 QA results for PR 886:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.cs.

[GitHub] spark pull request: [SPARK-2571] Correctly report shuffle read met...

2014-07-17 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/1476#discussion_r15098545 --- Diff: core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala --- @@ -81,8 +81,6 @@ class JobProgressListenerSuite extends FunSu

[GitHub] spark pull request: [SPARK-2359][MLlib] Correlations

2014-07-17 Thread dorx
Github user dorx commented on a diff in the pull request: https://github.com/apache/spark/pull/1367#discussion_r15098523 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmansCorrelation.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache S

[GitHub] spark pull request: [SPARK-2359][MLlib] Correlations

2014-07-17 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1367#discussion_r15098472 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmansCorrelation.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-2359][MLlib] Correlations

2014-07-17 Thread dorx
Github user dorx commented on a diff in the pull request: https://github.com/apache/spark/pull/1367#discussion_r15098446 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmansCorrelation.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache S

[GitHub] spark pull request: [SPARK-2359][MLlib] Correlations

2014-07-17 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1367#discussion_r15098421 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmansCorrelation.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: Fixed a typo in the comments in RangePartition...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1473#issuecomment-49400241 QA tests have started for PR 1473. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16816/consoleFull --- If

[GitHub] spark pull request: [SPARK-2359][MLlib] Correlations

2014-07-17 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1367#discussion_r15098382 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmansCorrelation.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-2359][MLlib] Correlations

2014-07-17 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1367#discussion_r15098384 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/stat/correlation/SpearmansCorrelation.scala --- @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: Reservoir sampling implementation.

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1478#issuecomment-49400124 QA results for PR 1478:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: Fixed a typo in the comments in RangePartition...

2014-07-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1473#discussion_r15098185 --- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala --- @@ -135,7 +135,7 @@ class RangePartitioner[K : Ordering : ClassTag, V]( val k = key.

[GitHub] spark pull request: Fixed a typo in the comments in RangePartition...

2014-07-17 Thread dorx
Github user dorx commented on a diff in the pull request: https://github.com/apache/spark/pull/1473#discussion_r15098158 --- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala --- @@ -135,7 +135,7 @@ class RangePartitioner[K : Ordering : ClassTag, V]( val k = key.

[GitHub] spark pull request: [SPARK-2570] [SQL] Fix the bug of ClassCastExc...

2014-07-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1475 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: Fixed a typo in the comments in RangePartition...

2014-07-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1473#discussion_r15098078 --- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala --- @@ -135,7 +135,7 @@ class RangePartitioner[K : Ordering : ClassTag, V]( val k = key.

[GitHub] spark pull request: [SPARK-2570] [SQL] Fix the bug of ClassCastExc...

2014-07-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1475#issuecomment-49399326 Merging. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark pull request: [SPARK-2359][MLlib] Correlations

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1367#issuecomment-49399247 QA tests have started for PR 1367. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16815/consoleFull --- If

[GitHub] spark pull request: [SPARK-2521] Broadcast RDD object (instead of ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1452#issuecomment-49399238 QA tests have started for PR 1452. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16814/consoleFull --- If

[GitHub] spark pull request: [SPARK-2571] Correctly report shuffle read met...

2014-07-17 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1476#discussion_r15097975 --- Diff: core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala --- @@ -81,8 +81,6 @@ class JobProgressListenerSuite extends FunSuite w

[GitHub] spark pull request: [SPARK-2521] Broadcast RDD object (instead of ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1452#issuecomment-49398792 QA tests have started for PR 1452. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16813/consoleFull --- If

[GitHub] spark pull request: put 'curRequestSize = 0' after 'logDebug' it

2014-07-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1477#issuecomment-49397603 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2571] Correctly report shuffle read met...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1476#issuecomment-49397551 QA tests have started for PR 1476. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16811/consoleFull --- If

[GitHub] spark pull request: Reservoir sampling implementation.

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1478#issuecomment-49396936 QA tests have started for PR 1478. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16810/consoleFull --- If

[GitHub] spark pull request: [MLlib] SPARK-1536: multiclass classification ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/886#issuecomment-49396292 QA tests have started for PR 886. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16809/consoleFull --- If y

[GitHub] spark pull request: [MLlib] SPARK-1536: multiclass classification ...

2014-07-17 Thread manishamde
Github user manishamde commented on the pull request: https://github.com/apache/spark/pull/886#issuecomment-49396157 @jkbradley Thanks! I just made another very minor commit to format MimaExcludes.scala. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: Reservoir sampling implementation.

2014-07-17 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1478#discussion_r15096878 --- Diff: core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala --- @@ -17,9 +17,49 @@ package org.apache.spark.util.random

[GitHub] spark pull request: [MLlib] SPARK-1536: multiclass classification ...

2014-07-17 Thread jkbradley
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/886#issuecomment-49395750 @mengxr LGTM. @manishamde Thanks for the fixes! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If you

[GitHub] spark pull request: Reservoir sampling implementation.

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1478#issuecomment-49395696 QA tests have started for PR 1478. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16808/consoleFull --- If

[GitHub] spark pull request: Reservoir sampling implementation.

2014-07-17 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1478 Reservoir sampling implementation. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark reservoirSample Alternatively you can review and app

[GitHub] spark pull request: [MLlib] SPARK-1536: multiclass classification ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/886#issuecomment-49395217 QA results for PR 886:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.cs.

[GitHub] spark pull request: [SPARK-2571] Correctly report shuffle read met...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1476#issuecomment-49394559 QA results for PR 1476:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2570] [SQL] Fix the bug of ClassCastExc...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1475#issuecomment-49393082 QA results for PR 1475:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: SPARK-2058: Overriding config from SPARK_HOME ...

2014-07-17 Thread lianhuiwang
Github user lianhuiwang commented on a diff in the pull request: https://github.com/apache/spark/pull/997#discussion_r15095873 --- Diff: bin/compute-classpath.sh --- @@ -30,6 +30,11 @@ FWDIR="$(cd `dirname $0`/..; pwd)" # Build up classpath CLASSPATH="$SPARK_CLASSPATH:$SPA

[GitHub] spark pull request: [SPARK-2125] Add sort flag and move sort into ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1210#issuecomment-49391815 QA results for PR 1210:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: put 'curRequestSize = 0' after 'logDebug' it

2014-07-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1477#issuecomment-49391628 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: put 'curRequestSize = 0' after 'logDebug' it

2014-07-17 Thread JerryLead
GitHub user JerryLead opened a pull request: https://github.com/apache/spark/pull/1477 put 'curRequestSize = 0' after 'logDebug' it This is a minor change. We should first logDebug($curRequestSize) and then set it to 0. You can merge this pull request into a Git repository by runni

[GitHub] spark pull request: Required AM memory is "amMem", not "args.amMem...

2014-07-17 Thread maji2014
Github user maji2014 commented on the pull request: https://github.com/apache/spark/pull/1457#issuecomment-49391500 Please focus on second issue as title. the first "Update run-example" is a old patch. --- If your project is set up for it, you can reply to this email and have your re

[GitHub] spark pull request: [MLlib] SPARK-1536: multiclass classification ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/886#issuecomment-49391450 QA tests have started for PR 886. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16807/consoleFull --- If y

[GitHub] spark pull request: [MLlib] SPARK-1536: multiclass classification ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/886#issuecomment-49391256 QA results for PR 886:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.cs.

[GitHub] spark pull request: [MLlib] SPARK-1536: multiclass classification ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/886#issuecomment-49391235 QA tests have started for PR 886. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16806/consoleFull --- If y

[GitHub] spark pull request: Required AM memory is "amMem", not "args.amMem...

2014-07-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1457#issuecomment-49391172 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: Required AM memory is "amMem", not "args.amMem...

2014-07-17 Thread maji2014
GitHub user maji2014 reopened a pull request: https://github.com/apache/spark/pull/1457 Required AM memory is "amMem", not "args.amMemory" "ERROR yarn.Client: Required AM memory (1024) is above the max threshold (1048) of this cluster" appears if this code is not changed. obviously,

[GitHub] spark pull request: The driver perform garbage collection, when th...

2014-07-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1387#issuecomment-49390933 @witgo please create a JIRA when proposing features like this. AFIAK the feature proposal is the following: if we detect memory pressure on the executors we shou

[GitHub] spark pull request: [SPARK-2571] Correctly report shuffle read met...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1476#issuecomment-49390789 QA tests have started for PR 1476. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16805/consoleFull --- If

[GitHub] spark pull request: Required AM memory is "amMem", not "args.amMem...

2014-07-17 Thread maji2014
Github user maji2014 closed the pull request at: https://github.com/apache/spark/pull/1457 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[GitHub] spark pull request: [Spark 2557] fix LOCAL_N_REGEX in createTaskSc...

2014-07-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1464#issuecomment-49390622 /cc @aarondav --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-2571] Correctly report shuffle read met...

2014-07-17 Thread kayousterhout
GitHub user kayousterhout opened a pull request: https://github.com/apache/spark/pull/1476 [SPARK-2571] Correctly report shuffle read metrics. Currently, shuffle read metrics are incorrectly reported when stages have multiple shuffle dependencies (they are set to be the metrics from

[GitHub] spark pull request: [SPARK-2542] Exit Code Class should be renamed...

2014-07-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1467#issuecomment-49390495 It doesn't seem obvious to me that we should rename this. The primary purpose here is to describe reasons why executors can fail. There is a slight corner case here wher

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

2014-07-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1336 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-2411] Add a history-not-found page to s...

2014-07-17 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1336#issuecomment-49390208 Okay cool - I'm merging tihs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request: SPARK-2380 [WIP]: Support displaying accumulat...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1309#issuecomment-49389878 QA results for PR 1309:- This patch FAILED unit tests.For more information see test ouptut:https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16804/consol

[GitHub] spark pull request: SPARK-2380 [WIP]: Support displaying accumulat...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1309#issuecomment-49389828 QA tests have started for PR 1309. This patch DID NOT merge cleanly! View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16804/consoleFull

[GitHub] spark pull request: Streaming mllib [SPARK-2438][MLLIB]

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1361#issuecomment-49389728 QA results for PR 1361:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2540] [SQL] Add HiveDecimal & HiveVarch...

2014-07-17 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/1436#issuecomment-49388750 @rxin I think this is ready to be merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your p

[GitHub] spark pull request: [SPARK-2570] [SQL] Fix the bug of ClassCastExc...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1475#issuecomment-49388650 QA tests have started for PR 1475. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16803/consoleFull --- If

[GitHub] spark pull request: [SPARK-2570] [SQL] Fix the bug of ClassCastExc...

2014-07-17 Thread chenghao-intel
GitHub user chenghao-intel opened a pull request: https://github.com/apache/spark/pull/1475 [SPARK-2570] [SQL] Fix the bug of ClassCastException Exception thrown when running the example of HiveFromSpark. Exception in thread "main" java.lang.ClassCastException: java.lang.Long ca

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-49388410 QA results for PR 1165:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds the following public classes (experimental):case class Sample(size: Lon

[GitHub] spark pull request: Fixed a typo in the comments in RangePartition...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1473#issuecomment-49388392 QA results for PR 1473:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2523] [SQL] [WIP] Hadoop table scan bug...

2014-07-17 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/1439#issuecomment-49388234 @yhuai I got your mean eventually, I think you're right, some of the logic could be shared among TableScan operators. --- If your project is set up for it, you ca

[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...

2014-07-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1262 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...

2014-07-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1262#issuecomment-49387996 Merging in master. Thanks for reviewing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-2564. ShuffleReadMetrics.totalBlocksRead...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1474#issuecomment-49387978 QA results for PR 1474:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1262#issuecomment-49387905 QA results for PR 1262:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2523] [SQL] [WIP] Hadoop table scan bug...

2014-07-17 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/1439#issuecomment-49387608 @yhuai sorry if I misundertood. Do you mean the `HiveTableScan` `ParquetTableScan` is the new operators, which created by SparkPlanner, right? --- If your project

[GitHub] spark pull request: [SPARK-2125] Add sort flag and move sort into ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1210#issuecomment-49387221 QA tests have started for PR 1210. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16801/consoleFull --- If

[GitHub] spark pull request: [SPARK-2523] [SQL] [WIP] Hadoop table scan bug...

2014-07-17 Thread yhuai
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/1439#issuecomment-49386783 @chenghao-intel I did not meant to introduce multiple `HiveTableScan`. I meant to have a abstract `TableScan` and make existing ones (e.g. `HiveTableScan` and `ParquetTable

[GitHub] spark pull request: [SPARK-2125] Add sort flag and move sort into ...

2014-07-17 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/1210#issuecomment-49386693 Shall we add this ``` ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.rdd.OrderedRDDFunctions.org$apache$spark$rdd$OrderedRDDFunctions$$orde

[GitHub] spark pull request: SPARK-2553. CoGroupedRDD unnecessarily allocat...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1461#issuecomment-49386170 QA results for PR 1461:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2523] [SQL] [WIP] Hadoop table scan bug...

2014-07-17 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/1439#issuecomment-49386103 @yhuai I agree with you we should make a clear boundary between `HiveTableScan` and `TableReader`, but I am not sure if it's a good idea to create multiple `HiveTa

[GitHub] spark pull request: [SPARK-2125] Add sort flag and move sort into ...

2014-07-17 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/1210#discussion_r15093470 --- Diff: core/src/main/scala/org/apache/spark/rdd/OrderedRDDFunctions.scala --- @@ -57,14 +57,13 @@ class OrderedRDDFunctions[K : Ordering : ClassTag,

[GitHub] spark pull request: SPARK-2553. CoGroupedRDD unnecessarily allocat...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1461#issuecomment-49386013 QA tests have started for PR 1461. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16800/consoleFull --- If

[GitHub] spark pull request: SPARK-2553. CoGroupedRDD unnecessarily allocat...

2014-07-17 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1461#issuecomment-49386002 Looks good to me, though this isn't avoiding *that* much since you are appending to an ArrayBuffer anyway. But no reason not to do it. --- If your project is set up for i

[GitHub] spark pull request: SPARK-2553. CoGroupedRDD unnecessarily allocat...

2014-07-17 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1461#issuecomment-49385926 Jenkins, add to whitelist and test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proj

[GitHub] spark pull request: [SPARK-2125] Add sort flag and move sort into ...

2014-07-17 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1210#issuecomment-49385778 @scrapcodes @pwendell there's a MIMA error on this that seems spurious: it complains that `synthetic method org$apache$spark$rdd$OrderedRDDFunctions$$ordering()scala.math.

[GitHub] spark pull request: [SPARK-2125] Add sort flag and move sort into ...

2014-07-17 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/1210#discussion_r15093402 --- Diff: core/src/main/scala/org/apache/spark/rdd/OrderedRDDFunctions.scala --- @@ -57,14 +57,13 @@ class OrderedRDDFunctions[K : Ordering : ClassTag,

[GitHub] spark pull request: [SPARK-2125] Add sort flag and move sort into ...

2014-07-17 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/1210#discussion_r15093320 --- Diff: core/src/main/scala/org/apache/spark/rdd/OrderedRDDFunctions.scala --- @@ -57,14 +57,13 @@ class OrderedRDDFunctions[K : Ordering : ClassTag,

[GitHub] spark pull request: Streaming mllib [SPARK-2438][MLLIB]

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1361#issuecomment-49384924 QA tests have started for PR 1361. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16799/consoleFull --- If

[GitHub] spark pull request: Streaming mllib [SPARK-2438][MLLIB]

2014-07-17 Thread freeman-lab
Github user freeman-lab commented on the pull request: https://github.com/apache/spark/pull/1361#issuecomment-49384594 @mengxr mind retesting? I tried to make the convergence test more robust in a couple ways. If we still have issues we might need to rethink that test further. Thanks!

[GitHub] spark pull request: [SPARK-2563] Make connection retries configura...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1471#issuecomment-49384160 QA results for PR 1471:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2523] [SQL] [WIP] Hadoop table scan bug...

2014-07-17 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1439#discussion_r15092708 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala --- @@ -156,33 +158,43 @@ class HadoopTableReader(@transient _tableDesc:

[GitHub] spark pull request: [SPARK-2523] [SQL] [WIP] Hadoop table scan bug...

2014-07-17 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1439#discussion_r15092652 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScan.scala --- @@ -67,95 +61,12 @@ case class HiveTableScan( }

[GitHub] spark pull request: SPARK-2564. ShuffleReadMetrics.totalBlocksRead...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1474#issuecomment-49383762 QA tests have started for PR 1474. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16798/consoleFull --- If

[GitHub] spark pull request: [SPARK-2454] Do not assume drivers and executo...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1472#issuecomment-49383563 QA results for PR 1472:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-49383527 Oops, the tests will fail. We can't just set an initial memory threshold for some reason. Investigating. --- If your project is set up for it, you can reply to this e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1165#issuecomment-49383461 QA tests have started for PR 1165. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16797/consoleFull --- If

[GitHub] spark pull request: SPARK-2564. ShuffleReadMetrics.totalBlocksRead...

2014-07-17 Thread sryza
GitHub user sryza opened a pull request: https://github.com/apache/spark/pull/1474 SPARK-2564. ShuffleReadMetrics.totalBlocksRead is redundant You can merge this pull request into a Git repository by running: $ git pull https://github.com/sryza/spark sandy-spark-2564 Alternat

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15092309 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: Fixed a typo in the comments in RangePartition...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1473#issuecomment-49383190 QA tests have started for PR 1473. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16796/consoleFull --- If

[GitHub] spark pull request: Fixed a typo in the comments in RangePartition...

2014-07-17 Thread dorx
GitHub user dorx opened a pull request: https://github.com/apache/spark/pull/1473 Fixed a typo in the comments in RangePartitioner Checked with Holden, the original author as per the log, and was told code is right comment is wrong. You can merge this pull request into a Git rep

[GitHub] spark pull request: [MLlib] SPARK-1536: multiclass classification ...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/886#issuecomment-49382897 QA results for PR 886:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.cs.

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15092083 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -140,14 +144,39 @@ private[spark] class CacheManager(blockManager: BlockManager) e

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15092024 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, max

[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...

2014-07-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1262#issuecomment-49382662 QA tests have started for PR 1262. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16795/consoleFull --- If

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15091852 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -463,16 +463,16 @@ private[spark] class BlockManager( v

[GitHub] spark pull request: SPARK-2407: Added Parser of SQL SUBSTR()

2014-07-17 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1442#issuecomment-49382058 I don't know how do it with maven but with sbt: `sbt/sbt "sql/test-only org.apache.spark.sql.SQLQuerySuite"` --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-1777] Prevent OOMs from single partitio...

2014-07-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1165#discussion_r15091634 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -141,6 +174,86 @@ private class MemoryStore(blockManager: BlockManager, max

[GitHub] spark pull request: SPARK-2407: Added Parser of SQL SUBSTR()

2014-07-17 Thread chutium
Github user chutium commented on the pull request: https://github.com/apache/spark/pull/1442#issuecomment-49381658 and... how can i run a single test class using maven? i tried: mvn -Pyarn -Phive -Dhadoop.version=2.3.0-cdh5.0.3 -Dtest=SQLQuerySuite test but it does not work --

[GitHub] spark pull request: [SPARK-2534] Avoid pulling in the entire RDD i...

2014-07-17 Thread rxin
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/1469 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: [SPARK-1667] Jobs never finish successfully on...

2014-07-17 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/1383#issuecomment-49379957 @rxin, I noticed some issues related to this issue. When, following 3 situation which maybe disk fault , executor doesn't stop. So, tasks assigned to the executor a

  1   2   3   >