git commit: [SQL] Pass SQLContext instead of SparkContext into physical operators.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.0 36668662f -> 1829ec411 [SQL] Pass SQLContext instead of SparkContext into physical operators. This makes it easier to use config options in operators. Author: Reynold Xin Closes #1164 from rxin/sqlcontext and squashes the following c

git commit: [SQL] Pass SQLContext instead of SparkContext into physical operators.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master 648553d48 -> ca5d8b590 [SQL] Pass SQLContext instead of SparkContext into physical operators. This makes it easier to use config options in operators. Author: Reynold Xin Closes #1164 from rxin/sqlcontext and squashes the following commi

git commit: Fix some tests.

2014-06-20 Thread pwendell
Repository: spark Updated Branches: refs/heads/master 010c460d6 -> 648553d48 Fix some tests. - JavaAPISuite was trying to compare a bare path with a URI. Fix by extracting the path from the URI, since we know it should be a local path anyway/ - b9be1609 excluded the ASM dependency everywh

git commit: [SPARK-2061] Made splits deprecated in JavaRDDLike

2014-06-20 Thread pwendell
Repository: spark Updated Branches: refs/heads/master a67864249 -> 010c460d6 [SPARK-2061] Made splits deprecated in JavaRDDLike The jira for the issue can be found at: https://issues.apache.org/jira/browse/SPARK-2061 Most of spark has used over to consistently using `partitions` instead of `

git commit: HOTFIX: Fixing style error introduced by 08d0ac

2014-06-20 Thread pwendell
Repository: spark Updated Branches: refs/heads/master e99903b84 -> a67864249 HOTFIX: Fixing style error introduced by 08d0ac Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a6786424 Tree: http://git-wip-us.apache.org/repos

git commit: [SPARK-1970] Update unit test in XORShiftRandomSuite to use ChiSquareTest from commons-math3

2014-06-20 Thread pwendell
Repository: spark Updated Branches: refs/heads/master 08d0aca78 -> e99903b84 [SPARK-1970] Update unit test in XORShiftRandomSuite to use ChiSquareTest from commons-math3 Updating the chisquare unit test in XORShiftRandomSuite to use the ChiSquareTest in commons-math3 instead of hardcoding th

git commit: SPARK-1902 Silence stacktrace from logs when doing port failover to port n+1

2014-06-20 Thread pwendell
Repository: spark Updated Branches: refs/heads/master 204478491 -> 08d0aca78 SPARK-1902 Silence stacktrace from logs when doing port failover to port n+1 Before: ``` 14/06/08 23:58:23 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already

git commit: [SQL] Use hive.SessionState, not the thread local SessionState

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.0 91dc0641c -> 36668662f [SQL] Use hive.SessionState, not the thread local SessionState Note that this is simply mimicing lookupRelation(). I do not have a concrete notion of why this solution is necessarily right-er than SessionState.ge

git commit: [SQL] Use hive.SessionState, not the thread local SessionState

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master d4c7572db -> 204478491 [SQL] Use hive.SessionState, not the thread local SessionState Note that this is simply mimicing lookupRelation(). I do not have a concrete notion of why this solution is necessarily right-er than SessionState.get, b

git commit: Move ScriptTransformation into the appropriate place.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.0 bb7141519 -> 91dc0641c Move ScriptTransformation into the appropriate place. Author: Reynold Xin Closes #1162 from rxin/script and squashes the following commits: 2c836b9 [Reynold Xin] Move ScriptTransformation into the appropriate p

git commit: Move ScriptTransformation into the appropriate place.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master 01125a116 -> d4c7572db Move ScriptTransformation into the appropriate place. Author: Reynold Xin Closes #1162 from rxin/script and squashes the following commits: 2c836b9 [Reynold Xin] Move ScriptTransformation into the appropriate place

git commit: Clean up CacheManager et al.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master 0ac71d128 -> 01125a116 Clean up CacheManager et al. **UPDATE** I have removed the special handling for `StorageLevel.MEMORY_*_SER` for now, because it introduces a potential performance regression. With the latest changes, this PR should

git commit: [SPARK-2225] Turn HAVING without GROUP BY into WHERE.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.0 c3cda49eb -> bb7141519 [SPARK-2225] Turn HAVING without GROUP BY into WHERE. @willb Author: Reynold Xin Closes #1161 from rxin/having-filter and squashes the following commits: fa8359a [Reynold Xin] [SPARK-2225] Turn HAVING without

git commit: [SPARK-2225] Turn HAVING without GROUP BY into WHERE.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master 171ebb3a8 -> 0ac71d128 [SPARK-2225] Turn HAVING without GROUP BY into WHERE. @willb Author: Reynold Xin Closes #1161 from rxin/having-filter and squashes the following commits: fa8359a [Reynold Xin] [SPARK-2225] Turn HAVING without GROU

git commit: SPARK-2180: support HAVING clauses in Hive queries

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.0 b1ea9e5bc -> c3cda49eb SPARK-2180: support HAVING clauses in Hive queries This PR extends Spark's HiveQL support to handle HAVING clauses in aggregations. The HAVING test from the Hive compatibility suite doesn't appear to be runnab

git commit: SPARK-2180: support HAVING clauses in Hive queries

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master 6a224c31e -> 171ebb3a8 SPARK-2180: support HAVING clauses in Hive queries This PR extends Spark's HiveQL support to handle HAVING clauses in aggregations. The HAVING test from the Hive compatibility suite doesn't appear to be runnable f

git commit: SPARK-1868: Users should be allowed to cogroup at least 4 RDDs

2014-06-20 Thread pwendell
Repository: spark Updated Branches: refs/heads/master d484ddeff -> 6a224c31e SPARK-1868: Users should be allowed to cogroup at least 4 RDDs Adds cogroup for 4 RDDs. Author: Allan Douglas R. de Oliveira Closes #813 from douglaz/more_cogroups and squashes the following commits: f8d6273 [Alla

git commit: [SPARK-2163] class LBFGS optimize with Double tolerance instead of Int

2014-06-20 Thread meng
Repository: spark Updated Branches: refs/heads/master 2f6a835e1 -> d484ddeff [SPARK-2163] class LBFGS optimize with Double tolerance instead of Int https://issues.apache.org/jira/browse/SPARK-2163 This pull request includes the change for **[SPARK-2163]**: * Changed the convergence tolerance

git commit: [SPARK-2163] class LBFGS optimize with Double tolerance instead of Int

2014-06-20 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.0 7594b3f94 -> b1ea9e5bc [SPARK-2163] class LBFGS optimize with Double tolerance instead of Int https://issues.apache.org/jira/browse/SPARK-2163 This pull request includes the change for **[SPARK-2163]**: * Changed the convergence toler

git commit: [SPARK-2218] rename Equals to EqualTo in Spark SQL expressions.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.0 0d05e13e0 -> 7594b3f94 [SPARK-2218] rename Equals to EqualTo in Spark SQL expressions. Due to the existence of scala.Equals, it is very error prone to name the expression Equals, especially because we use a lot of partial functions and

git commit: [SPARK-2218] rename Equals to EqualTo in Spark SQL expressions.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master 324952892 -> 2f6a835e1 [SPARK-2218] rename Equals to EqualTo in Spark SQL expressions. Due to the existence of scala.Equals, it is very error prone to name the expression Equals, especially because we use a lot of partial functions and pa

git commit: [SPARK-2209][SQL] Cast shouldn't do null check twice.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.0 a803e009c -> 86c4f5af9 [SPARK-2209][SQL] Cast shouldn't do null check twice. Also took the chance to clean up cast a little bit. Too many arrows on each line before! Author: Reynold Xin Closes #1143 from rxin/cast and squashes the f

git commit: [SPARK-2196] [SQL] Fix nullability of CaseWhen.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.0 86c4f5af9 -> 0d05e13e0 [SPARK-2196] [SQL] Fix nullability of CaseWhen. `CaseWhen` should use `branches.length` to check if `elseValue` is provided or not. Author: Takuya UESHIN Closes #1133 from ueshin/issues/SPARK-2196 and squashes

git commit: SPARK-2203: PySpark defaults to use same num reduce partitions as map side

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master c55bbb49f -> f46e02fcd SPARK-2203: PySpark defaults to use same num reduce partitions as map side For shuffle-based operators, such as rdd.groupBy() or rdd.sortByKey(), PySpark will always assume that the default parallelism to use for the

git commit: [SPARK-2196] [SQL] Fix nullability of CaseWhen.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master f46e02fcd -> 324952892 [SPARK-2196] [SQL] Fix nullability of CaseWhen. `CaseWhen` should use `branches.length` to check if `elseValue` is provided or not. Author: Takuya UESHIN Closes #1133 from ueshin/issues/SPARK-2196 and squashes the

git commit: [SPARK-2209][SQL] Cast shouldn't do null check twice.

2014-06-20 Thread rxin
Repository: spark Updated Branches: refs/heads/master 617564097 -> c55bbb49f [SPARK-2209][SQL] Cast shouldn't do null check twice. Also took the chance to clean up cast a little bit. Too many arrows on each line before! Author: Reynold Xin Closes #1143 from rxin/cast and squashes the follo