[1/2] spark git commit: [SPARK-15529][SQL] Replace SQLContext and HiveContext with SparkSession in Test

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master 6b1a6180e -> d5911d117 http://git-wip-us.apache.org/repos/asf/spark/blob/d5911d11/sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala -- diff --git

[1/2] spark git commit: [SPARK-15529][SQL] Replace SQLContext and HiveContext with SparkSession in Test

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 9c7e6ad28 -> b3845fede http://git-wip-us.apache.org/repos/asf/spark/blob/b3845fed/sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala -- diff --git

[2/2] spark git commit: [SPARK-15529][SQL] Replace SQLContext and HiveContext with SparkSession in Test

2016-05-26 Thread rxin
[SPARK-15529][SQL] Replace SQLContext and HiveContext with SparkSession in Test What changes were proposed in this pull request? This PR is to use the new entrance `Sparksession` to replace the existing `SQLContext` and `HiveContext` in SQL test suites. No change is made in the following

[2/2] spark git commit: [SPARK-15529][SQL] Replace SQLContext and HiveContext with SparkSession in Test

2016-05-26 Thread rxin
[SPARK-15529][SQL] Replace SQLContext and HiveContext with SparkSession in Test What changes were proposed in this pull request? This PR is to use the new entrance `Sparksession` to replace the existing `SQLContext` and `HiveContext` in SQL test suites. No change is made in the following

spark git commit: [MINOR] Fix Typos 'a -> an'

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master ee3609a2e -> 6b1a6180e [MINOR] Fix Typos 'a -> an' ## What changes were proposed in this pull request? `a` -> `an` I use regex to generate potential error lines: `grep -in ' a [aeiou]' mllib/src/main/scala/org/apache/spark/ml/*/*scala`

spark git commit: [MINOR] Fix Typos 'a -> an'

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 64d477cd4 -> 9c7e6ad28 [MINOR] Fix Typos 'a -> an' ## What changes were proposed in this pull request? `a` -> `an` I use regex to generate potential error lines: `grep -in ' a [aeiou]'

spark git commit: [MINOR][CORE] Fixed doc for Accumulator2.add

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master c82883239 -> ee3609a2e [MINOR][CORE] Fixed doc for Accumulator2.add ## What changes were proposed in this pull request? Scala doc used outdated ```+=```. Replaced with ```add```. ## How was this patch tested? N/A Author: Joseph K.

spark git commit: [MINOR][CORE] Fixed doc for Accumulator2.add

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 c1468447e -> 64d477cd4 [MINOR][CORE] Fixed doc for Accumulator2.add ## What changes were proposed in this pull request? Scala doc used outdated ```+=```. Replaced with ```add```. ## How was this patch tested? N/A Author: Joseph K.

spark git commit: [SPARK-10903] followup - update API doc for SqlContext

2016-05-26 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-2.0 4f66bf5fb -> c1468447e [SPARK-10903] followup - update API doc for SqlContext ## What changes were proposed in this pull request? Follow up on the earlier PR - in here we are fixing up roxygen2 doc examples. Also add to the

spark git commit: [SPARK-10903] followup - update API doc for SqlContext

2016-05-26 Thread shivaram
Repository: spark Updated Branches: refs/heads/master 1c403733b -> c82883239 [SPARK-10903] followup - update API doc for SqlContext ## What changes were proposed in this pull request? Follow up on the earlier PR - in here we are fixing up roxygen2 doc examples. Also add to the programming

spark git commit: [SPARK-8603][SPARKR] Use shell() instead of system2() for SparkR on Windows

2016-05-26 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-2.0 8e26b74fc -> 4f66bf5fb [SPARK-8603][SPARKR] Use shell() instead of system2() for SparkR on Windows ## What changes were proposed in this pull request? This PR corrects SparkR to use `shell()` instead of `system2()` on Windows. Using

spark git commit: [SPARK-8603][SPARKR] Use shell() instead of system2() for SparkR on Windows

2016-05-26 Thread shivaram
Repository: spark Updated Branches: refs/heads/master 3fca635b4 -> 1c403733b [SPARK-8603][SPARKR] Use shell() instead of system2() for SparkR on Windows ## What changes were proposed in this pull request? This PR corrects SparkR to use `shell()` instead of `system2()` on Windows. Using

spark git commit: [SPARK-15583][SQL] Disallow altering datasource properties

2016-05-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-2.0 702755f92 -> 8e26b74fc [SPARK-15583][SQL] Disallow altering datasource properties ## What changes were proposed in this pull request? Certain table properties (and SerDe properties) are in the protected namespace

spark git commit: [SPARK-15583][SQL] Disallow altering datasource properties

2016-05-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 6ab973ec5 -> 3fca635b4 [SPARK-15583][SQL] Disallow altering datasource properties ## What changes were proposed in this pull request? Certain table properties (and SerDe properties) are in the protected namespace `spark.sql.sources.`,

spark git commit: [SPARK-15542][SPARKR] Make error message clear for script './R/install-dev.sh' when R is missing on Mac

2016-05-26 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.0 c99ad9215 -> 702755f92 [SPARK-15542][SPARKR] Make error message clear for script './R/install-dev.sh' when R is missing on Mac https://issues.apache.org/jira/browse/SPARK-15542 ## What changes were proposed in this pull request?

spark git commit: [SPARK-15542][SPARKR] Make error message clear for script './R/install-dev.sh' when R is missing on Mac

2016-05-26 Thread srowen
Repository: spark Updated Branches: refs/heads/master 008a5377d -> 6ab973ec5 [SPARK-15542][SPARKR] Make error message clear for script './R/install-dev.sh' when R is missing on Mac https://issues.apache.org/jira/browse/SPARK-15542 ## What changes were proposed in this pull request? When

spark git commit: [SPARK-15538][SPARK-15539][SQL] Truncate table fixes round 2

2016-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-2.0 d3cd579d7 -> c99ad9215 [SPARK-15538][SPARK-15539][SQL] Truncate table fixes round 2 ## What changes were proposed in this pull request? Two more changes: (1) Fix truncate table for data source tables (only for cases without

spark git commit: [SPARK-15538][SPARK-15539][SQL] Truncate table fixes round 2

2016-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 3ac2363d7 -> 008a5377d [SPARK-15538][SPARK-15539][SQL] Truncate table fixes round 2 ## What changes were proposed in this pull request? Two more changes: (1) Fix truncate table for data source tables (only for cases without `PARTITION`)

spark git commit: [BUILD][1.6] Fix compilation

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.6 0b8bdf793 -> c53c83ce8 [BUILD][1.6] Fix compilation ## What changes were proposed in this pull request? Makes `UnsafeSortDataFormat` and `RecordPointerAndKeyPrefix` public. These are already public in 2.0 and are used in an

spark git commit: [SPARK-15532][SQL] SQLContext/HiveContext's public constructors should use SparkSession.build.getOrCreate

2016-05-26 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-2.0 4939c2a12 -> d3cd579d7 [SPARK-15532][SQL] SQLContext/HiveContext's public constructors should use SparkSession.build.getOrCreate ## What changes were proposed in this pull request? This PR changes SQLContext/HiveContext's public

spark git commit: [SPARK-15550][SQL] Dataset.show() should show contents nested products as rows

2016-05-26 Thread lian
Repository: spark Updated Branches: refs/heads/branch-2.0 29681cca1 -> 4939c2a12 [SPARK-15550][SQL] Dataset.show() should show contents nested products as rows ## What changes were proposed in this pull request? This PR addresses two related issues: 1. `Dataset.showString()` should show

spark git commit: [SPARK-15550][SQL] Dataset.show() should show contents nested products as rows

2016-05-26 Thread lian
Repository: spark Updated Branches: refs/heads/master fe6de16f7 -> e7082caeb [SPARK-15550][SQL] Dataset.show() should show contents nested products as rows ## What changes were proposed in this pull request? This PR addresses two related issues: 1. `Dataset.showString()` should show case

spark git commit: [SPARK-8428][SPARK-13850] Fix integer overflows in TimSort

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.6 5cc1e2cec -> 0b8bdf793 [SPARK-8428][SPARK-13850] Fix integer overflows in TimSort This patch fixes a few integer overflows in `UnsafeSortDataFormat.copyRange()` and `ShuffleSortDataFormat copyRange()` that seems to be the most likely

spark git commit: [SPARK-8428][SPARK-13850] Fix integer overflows in TimSort

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 7393449db -> 29681cca1 [SPARK-8428][SPARK-13850] Fix integer overflows in TimSort ## What changes were proposed in this pull request? This patch fixes a few integer overflows in `UnsafeSortDataFormat.copyRange()` and

spark git commit: [SPARK-8428][SPARK-13850] Fix integer overflows in TimSort

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master b5859e0bb -> fe6de16f7 [SPARK-8428][SPARK-13850] Fix integer overflows in TimSort ## What changes were proposed in this pull request? This patch fixes a few integer overflows in `UnsafeSortDataFormat.copyRange()` and

spark git commit: [SPARK-13445][SQL] Improves error message and add test coverage for Window function

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master b0a03feef -> b5859e0bb [SPARK-13445][SQL] Improves error message and add test coverage for Window function ## What changes were proposed in this pull request? Add more verbose error message when order by clause is missed when using

spark git commit: [SPARK-13445][SQL] Improves error message and add test coverage for Window function

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 216e39505 -> 7393449db [SPARK-13445][SQL] Improves error message and add test coverage for Window function ## What changes were proposed in this pull request? Add more verbose error message when order by clause is missed when using

spark git commit: [SPARK-15457][MLLIB][ML] Eliminate some warnings from MLlib about deprecations

2016-05-26 Thread jkbradley
Repository: spark Updated Branches: refs/heads/branch-2.0 6eea33ec3 -> 216e39505 [SPARK-15457][MLLIB][ML] Eliminate some warnings from MLlib about deprecations ## What changes were proposed in this pull request? Several classes and methods have been deprecated and are creating lots of build

spark git commit: [SPARK-15457][MLLIB][ML] Eliminate some warnings from MLlib about deprecations

2016-05-26 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 0f61d6efb -> b0a03feef [SPARK-15457][MLLIB][ML] Eliminate some warnings from MLlib about deprecations ## What changes were proposed in this pull request? Several classes and methods have been deprecated and are creating lots of build

spark git commit: [SPARK-15552][SQL] Remove unnecessary private[sql] methods in SparkSession

2016-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 594a1bf20 -> 0f61d6efb [SPARK-15552][SQL] Remove unnecessary private[sql] methods in SparkSession ## What changes were proposed in this pull request? SparkSession has a list of unnecessary private[sql] methods. These methods cause some

spark git commit: [SPARK-15520][SQL] Also set sparkContext confs when using SparkSession builder in pyspark

2016-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-2.0 69b3e9cee -> eb0c49799 [SPARK-15520][SQL] Also set sparkContext confs when using SparkSession builder in pyspark ## What changes were proposed in this pull request? Also sets confs in the underlying sc when using

spark git commit: [SPARK-15520][SQL] Also set sparkContext confs when using SparkSession builder in pyspark

2016-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 2b1ac6cea -> 594a1bf20 [SPARK-15520][SQL] Also set sparkContext confs when using SparkSession builder in pyspark ## What changes were proposed in this pull request? Also sets confs in the underlying sc when using

spark git commit: [SPARK-15539][SQL] DROP TABLE throw exception if table doesn't exist

2016-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-2.0 0cb69a918 -> 69b3e9cee [SPARK-15539][SQL] DROP TABLE throw exception if table doesn't exist ## What changes were proposed in this pull request? Same as #13302, but for DROP TABLE. ## How was this patch tested? `DDLSuite` Author:

spark git commit: [SPARK-15539][SQL] DROP TABLE throw exception if table doesn't exist

2016-05-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 01b350a4f -> 2b1ac6cea [SPARK-15539][SQL] DROP TABLE throw exception if table doesn't exist ## What changes were proposed in this pull request? Same as #13302, but for DROP TABLE. ## How was this patch tested? `DDLSuite` Author: Andrew

spark git commit: [SPARK-13148][YARN] document zero-keytab Oozie application launch; add diagnostics

2016-05-26 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-2.0 9cf34727c -> 0cb69a918 [SPARK-13148][YARN] document zero-keytab Oozie application launch; add diagnostics This patch provides detail on what to do for keytabless Oozie launches of spark apps, and adds some debug-level diagnostics of

spark git commit: [SPARK-13148][YARN] document zero-keytab Oozie application launch; add diagnostics

2016-05-26 Thread tgraves
Repository: spark Updated Branches: refs/heads/master c76457c8e -> 01b350a4f [SPARK-13148][YARN] document zero-keytab Oozie application launch; add diagnostics This patch provides detail on what to do for keytabless Oozie launches of spark apps, and adds some debug-level diagnostics of what

spark git commit: [SPARK-10903][SPARKR] R - Simplify SQLContext method signatures and use a singleton

2016-05-26 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-2.0 87374de43 -> 9cf34727c [SPARK-10903][SPARKR] R - Simplify SQLContext method signatures and use a singleton Eliminate the need to pass sqlContext to method since it is a singleton - and we don't want to support multiple contexts in a

svn commit: r1745631 [2/2] - in /spark: news/_posts/ site/ site/graphx/ site/mllib/ site/news/ site/releases/ site/screencasts/ site/sql/ site/streaming/

2016-05-26 Thread matei
Modified: spark/site/news/spark-summit-2014-videos-posted.html URL: http://svn.apache.org/viewvc/spark/site/news/spark-summit-2014-videos-posted.html?rev=1745631=1745630=1745631=diff == ---

spark git commit: [SPARK-10722] RDDBlockId not found in driver-heartbeater

2016-05-26 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.6 9a18115a8 -> 5cc1e2cec [SPARK-10722] RDDBlockId not found in driver-heartbeater ## What changes were proposed in this pull request? To ensure that the deserialization of TaskMetrics uses a ClassLoader that knows about RDDBlockIds.

spark git commit: [SPARK-15523][ML][MLLIB] Update JPMML to 1.2.15

2016-05-26 Thread srowen
Repository: spark Updated Branches: refs/heads/master e451f7f0c -> 6d506c9ae [SPARK-15523][ML][MLLIB] Update JPMML to 1.2.15 ## What changes were proposed in this pull request? See https://issues.apache.org/jira/browse/SPARK-15523 This PR replaces PR #13293. It's isolated to a new branch,

spark git commit: [SPARK-15523][ML][MLLIB] Update JPMML to 1.2.15

2016-05-26 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.0 c54a07348 -> 87374de43 [SPARK-15523][ML][MLLIB] Update JPMML to 1.2.15 ## What changes were proposed in this pull request? See https://issues.apache.org/jira/browse/SPARK-15523 This PR replaces PR #13293. It's isolated to a new

spark git commit: [SPARK-15492][ML][DOC] Binarization scala example copy & paste to spark-shell error

2016-05-26 Thread mlnick
Repository: spark Updated Branches: refs/heads/master 53d4abe9e -> e451f7f0c [SPARK-15492][ML][DOC] Binarization scala example copy & paste to spark-shell error ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) The Binarization scala example

spark git commit: [SPARK-15492][ML][DOC] Binarization scala example copy & paste to spark-shell error

2016-05-26 Thread mlnick
Repository: spark Updated Branches: refs/heads/branch-2.0 36acd53e8 -> c54a07348 [SPARK-15492][ML][DOC] Binarization scala example copy & paste to spark-shell error ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) The Binarization scala

spark git commit: [SPARK-15537][SQL] fix dir delete issue

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 b3ee53b84 -> 36acd53e8 [SPARK-15537][SQL] fix dir delete issue ## What changes were proposed in this pull request? For some of the test cases, e.g. `OrcSourceSuite`, it will create temp folders and temp files inside them. But after

spark git commit: [SPARK-15537][SQL] fix dir delete issue

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master 361ebc282 -> 53d4abe9e [SPARK-15537][SQL] fix dir delete issue ## What changes were proposed in this pull request? For some of the test cases, e.g. `OrcSourceSuite`, it will create temp folders and temp files inside them. But after tests

[1/4] spark git commit: [SPARK-15543][SQL] Rename DefaultSources to make them more self-describing

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 bcad1d13f -> b3ee53b84 http://git-wip-us.apache.org/repos/asf/spark/blob/b3ee53b8/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala -- diff --git

[3/4] spark git commit: [SPARK-15543][SQL] Rename DefaultSources to make them more self-describing

2016-05-26 Thread rxin
http://git-wip-us.apache.org/repos/asf/spark/blob/361ebc28/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala -- diff --git

[1/4] spark git commit: [SPARK-15543][SQL] Rename DefaultSources to make them more self-describing

2016-05-26 Thread rxin
Repository: spark Updated Branches: refs/heads/master dfc9fc02c -> 361ebc282 http://git-wip-us.apache.org/repos/asf/spark/blob/361ebc28/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala -- diff --git

[2/4] spark git commit: [SPARK-15543][SQL] Rename DefaultSources to make them more self-describing

2016-05-26 Thread rxin
http://git-wip-us.apache.org/repos/asf/spark/blob/b3ee53b8/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/DefaultSource.scala -- diff --git

[4/4] spark git commit: [SPARK-15543][SQL] Rename DefaultSources to make them more self-describing

2016-05-26 Thread rxin
[SPARK-15543][SQL] Rename DefaultSources to make them more self-describing ## What changes were proposed in this pull request? This patch renames various DefaultSources to make their names more self-describing. The choice of "DefaultSource" was from the days when we did not have a good way to

[3/4] spark git commit: [SPARK-15543][SQL] Rename DefaultSources to make them more self-describing

2016-05-26 Thread rxin
http://git-wip-us.apache.org/repos/asf/spark/blob/b3ee53b8/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala -- diff --git

[4/4] spark git commit: [SPARK-15543][SQL] Rename DefaultSources to make them more self-describing

2016-05-26 Thread rxin
[SPARK-15543][SQL] Rename DefaultSources to make them more self-describing ## What changes were proposed in this pull request? This patch renames various DefaultSources to make their names more self-describing. The choice of "DefaultSource" was from the days when we did not have a good way to

[2/4] spark git commit: [SPARK-15543][SQL] Rename DefaultSources to make them more self-describing

2016-05-26 Thread rxin
http://git-wip-us.apache.org/repos/asf/spark/blob/361ebc28/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/DefaultSource.scala -- diff --git