[GitHub] spark pull request: [SPARK-2995][MLLIB] add ALS.setIntermediateRDD...

2014-08-13 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1913#issuecomment-52150073 Merged into both master and branch-1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark pull request: [SPARK-2873] [SQL] using ExternalAppendOnlyMap...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1822#discussion_r16223599 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala --- @@ -292,6 +312,11 @@ case class AverageFunction(e

[GitHub] spark pull request: Added support for :cp that was broken in...

2014-08-13 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/1929#issuecomment-52149882 Looks okay to me, I was wondering if you tried scala reflection before resorting to java reflection ? --- If your project is set up for it, you can reply to this emai

[GitHub] spark pull request: [SPARK-2995][MLLIB] add ALS.setIntermediateRDD...

2014-08-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1913 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: SPARK-1170 Added histogram(buckets) to pyspark...

2014-08-13 Thread ScrapCodes
Github user ScrapCodes closed the pull request at: https://github.com/apache/spark/pull/121 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[GitHub] spark pull request: [SPARK-3015] Block on cleaning tasks to preven...

2014-08-13 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1931#issuecomment-52149487 Its a little ugly that the ContextCleaner class is being polluted with so many parameters, and all the temporary queue length code. Wouldnt it be much cleaner if we make a c

[GitHub] spark pull request: [SPARK-2979][MLlib] Improve the convergence ra...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1897#issuecomment-52149464 QA tests have started for PR 1897. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18527/consoleFull --- If

[GitHub] spark pull request: [SPARK-3029] Disable local execution of Spark ...

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1321#issuecomment-52149427 You need to update the test suites. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2759][CORE] Generic Binary File Support...

2014-08-13 Thread freeman-lab
Github user freeman-lab commented on the pull request: https://github.com/apache/spark/pull/1658#issuecomment-52149331 @kmader great to incorporate the FixedRecord stuff into the PR, thanks! I like @mateiz's suggestion for naming the two methods. I was starting to work on ``saveAsBina

[GitHub] spark pull request: [SPARK-2979][MLlib] Improve the convergence ra...

2014-08-13 Thread dbtsai
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/1897#issuecomment-52149162 Seems that Jenkins is not stable. Failing on issues related to akka. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-3029] Disable local execution of Spark ...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1321#issuecomment-52149122 QA results for PR 1321:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2979][MLlib] Improve the convergence ra...

2014-08-13 Thread dbtsai
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/1897#issuecomment-52149135 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-3015] Block on cleaning tasks to preven...

2014-08-13 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1931#discussion_r16223115 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -65,11 +66,37 @@ private[spark] class ContextCleaner(sc: SparkContext) extends Logging

[GitHub] spark pull request: [SPARK-2873] [SQL] using ExternalAppendOnlyMap...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1822#discussion_r16223080 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala --- @@ -114,6 +115,14 @@ case class MinFunction(expr:

[GitHub] spark pull request: [SPARK-2873] [SQL] using ExternalAppendOnlyMap...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1822#discussion_r16223044 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala --- @@ -375,6 +420,10 @@ case class SumDistinctFuncti

[GitHub] spark pull request: [SPARK-2873] [SQL] using ExternalAppendOnlyMap...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1822#discussion_r16222998 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala --- @@ -368,6 +411,8 @@ case class SumDistinctFunctio

[GitHub] spark pull request: [SPARK-2995][MLLIB] add ALS.setIntermediateRDD...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1913#issuecomment-52148676 QA results for PR 1913:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2979][MLlib] Improve the convergence ra...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1897#issuecomment-52148128 QA results for PR 1897:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-3015] Block on cleaning tasks to preven...

2014-08-13 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1931#discussion_r16222847 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -171,12 +203,44 @@ private[spark] class ContextCleaner(sc: SparkContext) extends Loggi

[GitHub] spark pull request: [SPARK-3001][MLLIB] Improve Spearman's correla...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1917#issuecomment-52148077 QA results for PR 1917:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-3015] Block on cleaning tasks to preven...

2014-08-13 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1931#discussion_r16222813 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -65,11 +66,37 @@ private[spark] class ContextCleaner(sc: SparkContext) extends Logging

[GitHub] spark pull request: [Docs] Add missing tags (minor)

2014-08-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1936 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-3011][SQL] _temporary directory should ...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1924#issuecomment-52147949 QA tests have started for PR 1924. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18526/consoleFull --- If

[GitHub] spark pull request: [SPARK-3015] Block on cleaning tasks to preven...

2014-08-13 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1931#discussion_r16222794 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -171,12 +203,44 @@ private[spark] class ContextCleaner(sc: SparkContext) extends Loggi

[GitHub] spark pull request: [SPARK-3027] TaskContext: tighten visibility a...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1938#issuecomment-52147942 QA tests have started for PR 1938. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18525/consoleFull --- If

[GitHub] spark pull request: [Docs] Add missing tags (minor)

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1936#issuecomment-52147906 Thanks. Merging in master & branch-1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-3027] TaskContext: tighten the visibili...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1938#issuecomment-52147699 QA results for PR 1938:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds the following public classes (experimental):trait TaskCompletionListene

[GitHub] spark pull request: [SPARK-3027] TaskContext: tighten the visibili...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1938#issuecomment-52147690 QA tests have started for PR 1938. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18524/consoleFull --- If

[GitHub] spark pull request: [SPARK-3011][SQL] _temporary directory should ...

2014-08-13 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1924#issuecomment-52147673 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featu

[GitHub] spark pull request: [SPARK-3011][SQL] _temporary directory should ...

2014-08-13 Thread joesu
Github user joesu commented on the pull request: https://github.com/apache/spark/pull/1924#issuecomment-52147641 code fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enab

[GitHub] spark pull request: [SPARK-3027] TaskContext: tighten the visibili...

2014-08-13 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1938 [SPARK-3027] TaskContext: tighten the visibility and provide Java friendly callback API You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark

[GitHub] spark pull request: SPARK-3025: Allow JDBC clients to set a fair s...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1937#issuecomment-52147446 QA results for PR 1937:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [MLlib] Word2Vec performance improvement

2014-08-13 Thread Ishiihara
Github user Ishiihara commented on a diff in the pull request: https://github.com/apache/spark/pull/1932#discussion_r16222465 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala --- @@ -284,16 +284,15 @@ class Word2Vec extends Serializable with Logging {

[GitHub] spark pull request: [HOTFIX][STREAMING] Allow the JVM/Netty to dec...

2014-08-13 Thread harishreedharan
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/1820#issuecomment-52146776 Can one of the committers please merge this? It would be nice to have this pulled in --- If your project is set up for it, you can reply to this email and have y

[GitHub] spark pull request: [RFC] Disable local execution of Spark jobs by...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1321#issuecomment-52146688 QA tests have started for PR 1321. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18523/consoleFull --- If

[GitHub] spark pull request: [MLlib] Word2Vec performance improvement

2014-08-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1932#discussion_r1600 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala --- @@ -284,16 +284,15 @@ class Word2Vec extends Serializable with Logging {

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-13 Thread colorant
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52146432 @mateiz rebased to latest master code. Have time to take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] spark pull request: [RFC] Disable local execution of Spark jobs by...

2014-08-13 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1321#issuecomment-52146322 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [RFC] Disable local execution of Spark jobs by...

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1321#issuecomment-52146296 Now I think about it more. LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark pull request: [RFC] Disable local execution of Spark jobs by...

2014-08-13 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1321#issuecomment-52146302 :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and w

[GitHub] spark pull request: [MLlib] Word2Vec performance improvement

2014-08-13 Thread Ishiihara
Github user Ishiihara commented on a diff in the pull request: https://github.com/apache/spark/pull/1932#discussion_r16222127 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala --- @@ -284,16 +284,15 @@ class Word2Vec extends Serializable with Logging {

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/1935#issuecomment-52146130 Don't forget the `SumDistinct`, :-) One concern is about the memory usage after partition data shuffled. e.g. `select sum(distinct(value)) from src`

[GitHub] spark pull request: [SPARK-2995][MLLIB] add ALS.setIntermediateDat...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1913#issuecomment-52145955 QA tests have started for PR 1913. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18522/consoleFull --- If

[GitHub] spark pull request: [SPARK-3001][MLLIB] Improve Spearman's correla...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1917#issuecomment-52145718 QA tests have started for PR 1917. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18520/consoleFull --- If

[GitHub] spark pull request: [SPARK-2979][MLlib] Improve the convergence ra...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1897#issuecomment-52145716 QA tests have started for PR 1897. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18521/consoleFull --- If

[GitHub] spark pull request: [SPARK-2995][MLLIB] add ALS.setIntermediateDat...

2014-08-13 Thread brkyvz
Github user brkyvz commented on the pull request: https://github.com/apache/spark/pull/1913#issuecomment-52145683 @mengxr: I would prefer setIntermediateRDDStorageLevel. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1935#discussion_r16221878 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -280,7 +280,7 @@ abstract class TreeNode[BaseType <: TreeNod

[GitHub] spark pull request: [Docs] Add missing tags (minor)

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1936#issuecomment-52145558 QA results for PR 1936:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-3001][MLLIB] Improve Spearman's correla...

2014-08-13 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1917#issuecomment-52145532 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-2979][MLlib] Improve the convergence ra...

2014-08-13 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1897#issuecomment-52145394 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-2979][MLlib] Improve the convergence ra...

2014-08-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1897#discussion_r16221810 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/regression/GeneralizedLinearAlgorithm.scala --- @@ -137,11 +154,45 @@ abstract class GeneralizedLinearAl

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/1935#discussion_r16221803 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala --- @@ -65,6 +65,80 @@ abstract class PartialAggregate exte

[GitHub] spark pull request: [MLlib] Word2Vec performance improvement

2014-08-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1932#discussion_r16221793 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala --- @@ -321,42 +320,46 @@ class Word2Vec extends Serializable with Logging {

[GitHub] spark pull request: [MLlib] Word2Vec performance improvement

2014-08-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1932#discussion_r16221788 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala --- @@ -284,16 +284,15 @@ class Word2Vec extends Serializable with Logging {

[GitHub] spark pull request: [MLlib] Word2Vec performance improvement

2014-08-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1932#discussion_r16221784 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala --- @@ -34,7 +34,7 @@ import org.apache.spark.mllib.rdd.RDDFunctions._ import

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1935#discussion_r16221796 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala --- @@ -65,6 +65,80 @@ abstract class PartialAggregat

[GitHub] spark pull request: [MLlib] Word2Vec performance improvement

2014-08-13 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1932#discussion_r16221791 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala --- @@ -321,42 +320,46 @@ class Word2Vec extends Serializable with Logging {

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1935#discussion_r16221759 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala --- @@ -65,6 +65,80 @@ abstract class PartialAggregat

[GitHub] spark pull request: [MLlib] Word2Vec performance improvement

2014-08-13 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1932#issuecomment-52144922 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1935#discussion_r16221620 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -280,7 +280,7 @@ abstract class TreeNode[BaseType <: T

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1935#discussion_r16221600 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -280,7 +280,7 @@ abstract class TreeNode[BaseType <: T

[GitHub] spark pull request: [SPARK-3006] Failed to execute spark-shell in ...

2014-08-13 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1918 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-3006] Failed to execute spark-shell in ...

2014-08-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1918#issuecomment-52144740 Thanks, merged into master and branch-1.1! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proj

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on a diff in the pull request: https://github.com/apache/spark/pull/1935#discussion_r16221450 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala --- @@ -65,6 +65,80 @@ abstract class PartialAggregat

[GitHub] spark pull request: SPARK-3025: Allow JDBC clients to set a fair s...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1937#issuecomment-52144396 QA tests have started for PR 1937. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18517/consoleFull --- If

[GitHub] spark pull request: SPARK-3025: Allow JDBC clients to set a fair s...

2014-08-13 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/1937 SPARK-3025: Allow JDBC clients to set a fair scheduler pool This definitely needs review as I am not familiar with this part of Spark. I tested this locally and it did seem to work. You can mer

[GitHub] spark pull request: [SPARK-2169] Don't copy appName / basePath eve...

2014-08-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1252#issuecomment-52143479 Looks great. Thanks @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [Docs] Add missing tags (minor)

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1936#issuecomment-52143339 QA tests have started for PR 1936. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18516/consoleFull --- If

[GitHub] spark pull request: [Docs] Add missing tags (minor)

2014-08-13 Thread andrewor14
GitHub user andrewor14 opened a pull request: https://github.com/apache/spark/pull/1936 [Docs] Add missing tags (minor) These configs looked inconsistent from the rest. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewor14/s

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1935#issuecomment-52141677 QA results for PR 1935:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds the following public classes (experimental):abstract class MergeableAgg

[GitHub] spark pull request: [SPARK-3007][SQL]Add "Dynamic Partition" suppo...

2014-08-13 Thread baishuo
Github user baishuo commented on the pull request: https://github.com/apache/spark/pull/1919#issuecomment-52141005 hi @marmbrus , when I study the HiveQuerySuite.scala, I found there is a important table : src, but I didnt find where and how the table created, would please give more

[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-52139906 QA tests have started for PR 1241. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18515/consoleFull --- If

[GitHub] spark pull request: [SPARK-2886] Use more specific actor system na...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1810#issuecomment-52139774 QA results for PR 1810:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52139272 QA results for PR 1907:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: SPARK-1719: spark.*.extraLibraryPath isn't app...

2014-08-13 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1022#issuecomment-52138943 YARN does not seem to do any processing. We can use the solution in #1031 --- If your project is set up for it, you can reply to this email and have your reply appear on G

[GitHub] spark pull request: [SPARK-3003] FailedStage could not be cancelle...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1921#issuecomment-52138727 QA results for PR 1921:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2917] [SQL] Avoid table creation in log...

2014-08-13 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/1846#issuecomment-52138495 Still failed in pyspark. But it passed in my local. Any suggestion? ``` = Running

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1935#issuecomment-52138366 QA tests have started for PR 1935. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18514/consoleFull --- If

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-13 Thread marmbrus
GitHub user marmbrus opened a pull request: https://github.com/apache/spark/pull/1935 [WIP][SPARK-2554][SQL] CountDistinct and SumDistinct should do partial aggregation You can merge this pull request into a Git repository by running: $ git pull https://github.com/marmbrus/sp

[GitHub] spark pull request: SPARK-3014. Log a more informative messages in...

2014-08-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1934#issuecomment-52137483 Tests actually passed, just not posted here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-2886] Use more specific actor system na...

2014-08-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1810#issuecomment-52137127 The newest changes renames the actor system to `sparkDriver` and `sparkExecutor`. The scope of the changes has grown a little, as it turns out we currently hard-code t

[GitHub] spark pull request: SPARK-2333

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1899#issuecomment-52137075 QA results for PR 1899:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2886] Use more specific actor system na...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1810#issuecomment-52136993 QA tests have started for PR 1810. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18513/consoleFull --- If

[GitHub] spark pull request: [SPARK-2917] [SQL] Avoid table creation in log...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1846#issuecomment-52136949 QA results for PR 1846:- This patch FAILED unit tests.- This patch merges cleanly- This patch adds the following public classes (experimental):case class CreateTableAsSel

[GitHub] spark pull request: [SPARK-2886] Use more specific actor system na...

2014-08-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1810#discussion_r16218675 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisorImpl.scala --- @@ -20,22 +20,21 @@ package org.apache.spark.strea

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52136472 QA tests have started for PR 1907. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18512/consoleFull --- If

[GitHub] spark pull request: [SPARK-3015] Block on cleaning tasks to preven...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1931#issuecomment-52136264 QA results for PR 1931:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2927][SQL] Add a conf to configure if w...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1855#issuecomment-52136157 QA results for PR 1855:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2886] Use more specific actor system na...

2014-08-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1810#discussion_r16218351 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -146,9 +146,9 @@ object SparkEnv extends Logging { } val securi

[GitHub] spark pull request: [SPARK-3003] FailedStage could not be cancelle...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1921#issuecomment-52135972 QA tests have started for PR 1921. This patch merges cleanly. View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18511/consoleFull --- If

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16218268 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -262,77 +270,51 @@ object BlockFetcherIterator { readMetrics:

[GitHub] spark pull request: [SPARK-2927][SQL] Add a conf to configure if w...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1855#issuecomment-52135873 QA results for PR 1855:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2924] remove default args to overloaded...

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1704#issuecomment-52135649 QA results for PR 1704:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-2925] [sql]fix spark-sql and start-thri...

2014-08-13 Thread scwf
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/1851#issuecomment-52135625 @marmbrus, the test failures but it seems error when running PySpark tests. And the jenkins seems not post tests result to Github. Can you suggest for this? --- If yo

[GitHub] spark pull request: [SQL] Python JsonRDD UTF8 Encoding Fix

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1914#issuecomment-52135573 QA results for PR 1914:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: SPARK-2333

2014-08-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1899#issuecomment-52135326 QA results for PR 1899:- This patch PASSES unit tests.- This patch merges cleanly- This patch adds no public classesFor more information see test ouptut:https://amplab.c

[GitHub] spark pull request: [SPARK-3006] Failed to execute spark-shell in ...

2014-08-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1918#issuecomment-52134918 Doing the equivalent in #1825 is more involved and we may do that some time down the road. This is fine as is for now. LGTM --- If your project is set up for it, you

[GitHub] spark pull request: SPARK-2900. aggregate inputBytes per stage

2014-08-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1826#issuecomment-52134705 Looks like this code was simply missing for input bytes. LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] spark pull request: SPARK-2900. aggregate inputBytes per stage

2014-08-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1826#discussion_r16217740 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala --- @@ -199,6 +199,12 @@ class JobProgressListener(conf: SparkConf) extend

[GitHub] spark pull request: [SPARK-3006] Failed to execute spark-shell in ...

2014-08-13 Thread tsudukim
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1918#issuecomment-52134614 Thanks @andrewor14 to follow this PR. You're right so i modified to put {{%*}} before {{spark-shell}}. but application arguments are not available yet until we make c

  1   2   3   4   5   6   >