spark git commit: [SPARK-17242][DOCUMENT] Update links of external dstream projects

2016-08-25 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-2.0 73014a2aa -> 27ed6d5dc [SPARK-17242][DOCUMENT] Update links of external dstream projects ## What changes were proposed in this pull request? Updated links of external dstream projects. ## How was this patch tested? Just document

spark git commit: [SPARK-17242][DOCUMENT] Update links of external dstream projects

2016-08-25 Thread rxin
Repository: spark Updated Branches: refs/heads/master b964a172a -> 341e0e778 [SPARK-17242][DOCUMENT] Update links of external dstream projects ## What changes were proposed in this pull request? Updated links of external dstream projects. ## How was this patch tested? Just document

spark git commit: [SPARK-17212][SQL] TypeCoercion supports widening conversion between DateType and TimestampType

2016-08-25 Thread wenchen
Repository: spark Updated Branches: refs/heads/master d96d15156 -> b964a172a [SPARK-17212][SQL] TypeCoercion supports widening conversion between DateType and TimestampType ## What changes were proposed in this pull request? Currently, type-widening does not work between `TimestampType` and

spark git commit: [SPARK-17187][SQL] Supports using arbitrary Java object as internal aggregation buffer object

2016-08-25 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 9b5a1d1d5 -> d96d15156 [SPARK-17187][SQL] Supports using arbitrary Java object as internal aggregation buffer object ## What changes were proposed in this pull request? This PR introduces an abstract class `TypedImperativeAggregate` so

spark git commit: [SPARK-17231][CORE] Avoid building debug or trace log messages unless the respective log level is enabled

2016-08-25 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.0 ff2e270eb -> 73014a2aa [SPARK-17231][CORE] Avoid building debug or trace log messages unless the respective log level is enabled This is simply a backport of #14798 to `branch-2.0`. This backport omits the change to

spark git commit: [SPARK-17240][CORE] Make SparkConf serializable again.

2016-08-25 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 3e4c7db4d -> 9b5a1d1d5 [SPARK-17240][CORE] Make SparkConf serializable again. Make the config reader transient, and initialize it lazily so that serialization works with both java and kryo (and hopefully any other custom serializer).

spark git commit: [SPARK-17205] Literal.sql should handle Infinity and NaN

2016-08-25 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 b3a44306a -> ff2e270eb [SPARK-17205] Literal.sql should handle Infinity and NaN This patch updates `Literal.sql` to properly generate SQL for `NaN` and `Infinity` float and double literals: these special values need to be handled

spark git commit: [SPARK-17205] Literal.sql should handle Infinity and NaN

2016-08-25 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master a133057ce -> 3e4c7db4d [SPARK-17205] Literal.sql should handle Infinity and NaN This patch updates `Literal.sql` to properly generate SQL for `NaN` and `Infinity` float and double literals: these special values need to be handled

spark git commit: [SPARK-17229][SQL] PostgresDialect shouldn't widen float and short types during reads

2016-08-25 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 9958ac0ce -> a133057ce [SPARK-17229][SQL] PostgresDialect shouldn't widen float and short types during reads ## What changes were proposed in this pull request? When reading float4 and smallint columns from PostgreSQL, Spark's

spark git commit: [SPARKR][BUILD] ignore cran-check.out under R folder

2016-08-25 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-2.0 55db26245 -> b3a44306a [SPARKR][BUILD] ignore cran-check.out under R folder ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) R add cran check which will generate the cran-check.out.

spark git commit: [SPARK-17231][CORE] Avoid building debug or trace log messages unless the respective log level is enabled

2016-08-25 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master d2ae6399e -> f20931071 [SPARK-17231][CORE] Avoid building debug or trace log messages unless the respective log level is enabled (This PR addresses https://issues.apache.org/jira/browse/SPARK-17231) ## What changes were proposed in this

spark git commit: [SPARK-16700][PYSPARK][SQL] create DataFrame from dict/Row with schema

2016-08-25 Thread davies
Repository: spark Updated Branches: refs/heads/branch-2.0 2b32a442d -> 356a359de [SPARK-16700][PYSPARK][SQL] create DataFrame from dict/Row with schema In 2.0, we verify the data type against schema for every row for safety, but with performance cost, this PR make it optional. When we

spark git commit: [SPARK-16991][SPARK-17099][SPARK-17120][SQL] Fix Outer Join Elimination when Filter's isNotNull Constraints Unable to Filter Out All Null-supplying Rows

2016-08-25 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 184e78b9d -> 48ecf3d00 [SPARK-16991][SPARK-17099][SPARK-17120][SQL] Fix Outer Join Elimination when Filter's isNotNull Constraints Unable to Filter Out All Null-supplying Rows ### What changes were proposed in this pull request? This

spark git commit: [SPARK-17061][SPARK-17093][SQL][BACKPORT] MapObjects should make copies of unsafe-backed data

2016-08-25 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 88481ea21 -> 184e78b9d [SPARK-17061][SPARK-17093][SQL][BACKPORT] MapObjects should make copies of unsafe-backed data ## What changes were proposed in this pull request? This PR backports https://github.com/apache/spark/pull/14698 to

spark git commit: [SPARK-12978][SQL] Skip unnecessary final group-by when input data already clustered with group-by keys

2016-08-25 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 6b8cb1fe5 -> 2b0cc4e0d [SPARK-12978][SQL] Skip unnecessary final group-by when input data already clustered with group-by keys This ticket targets the optimization to skip an unnecessary group-by operation below; Without opt.: ``` ==

spark git commit: Revert "[SPARK-17061][SPARK-17093][SQL] MapObjects` should make copies of unsafe-backed data"

2016-08-25 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 fb1c69714 -> 88481ea21 Revert "[SPARK-17061][SPARK-17093][SQL] MapObjects` should make copies of unsafe-backed data" This reverts commit fb1c697143a5bb2df69d9f2c9cbddc4eb526f047. Project:

spark git commit: [SPARK-17061][SPARK-17093][SQL] MapObjects` should make copies of unsafe-backed data

2016-08-25 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/branch-2.0 c1c498006 -> fb1c69714 [SPARK-17061][SPARK-17093][SQL] MapObjects` should make copies of unsafe-backed data Currently `MapObjects` does not make copies of unsafe-backed data, leading to problems like

spark git commit: [SPARK-17197][ML][PYSPARK] PySpark LiR/LoR supports tree aggregation level configurable.

2016-08-25 Thread yliang
Repository: spark Updated Branches: refs/heads/master e0b20f9f2 -> 6b8cb1fe5 [SPARK-17197][ML][PYSPARK] PySpark LiR/LoR supports tree aggregation level configurable. ## What changes were proposed in this pull request? [SPARK-17090](https://issues.apache.org/jira/browse/SPARK-17090) makes

spark git commit: [SPARK-17061][SPARK-17093][SQL] MapObjects` should make copies of unsafe-backed data

2016-08-25 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 2bcd5d5ce -> e0b20f9f2 [SPARK-17061][SPARK-17093][SQL] MapObjects` should make copies of unsafe-backed data ## What changes were proposed in this pull request? Currently `MapObjects` does not make copies of unsafe-backed data, leading to

spark git commit: [SPARK-17193][CORE] HadoopRDD NPE at DEBUG log level when getLocationInfo == null

2016-08-25 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.0 aa57083af -> c1c498006 [SPARK-17193][CORE] HadoopRDD NPE at DEBUG log level when getLocationInfo == null ## What changes were proposed in this pull request? Handle null from Hadoop getLocationInfo directly instead of catching (and

spark git commit: [SPARK-17193][CORE] HadoopRDD NPE at DEBUG log level when getLocationInfo == null

2016-08-25 Thread srowen
Repository: spark Updated Branches: refs/heads/master 5f02d2e5b -> 2bcd5d5ce [SPARK-17193][CORE] HadoopRDD NPE at DEBUG log level when getLocationInfo == null ## What changes were proposed in this pull request? Handle null from Hadoop getLocationInfo directly instead of catching (and

spark git commit: [SPARK-17215][SQL] Method `SQLContext.parseDataType(dataTypeString: String)` could be removed.

2016-08-25 Thread rxin
Repository: spark Updated Branches: refs/heads/master 4d0706d61 -> 5f02d2e5b [SPARK-17215][SQL] Method `SQLContext.parseDataType(dataTypeString: String)` could be removed. ## What changes were proposed in this pull request? Method `SQLContext.parseDataType(dataTypeString: String)` could be