spark git commit: [SPARK-14149] Log exceptions in tryOrIOException

2016-03-25 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.5 6eec04e0a -> 1bba62e93 [SPARK-14149] Log exceptions in tryOrIOException ## What changes were proposed in this pull request? We ran into a problem today debugging some class loading problem during deserialization, and JVM was masking th

spark git commit: [SPARK-14149] Log exceptions in tryOrIOException

2016-03-25 Thread rxin
Repository: spark Updated Branches: refs/heads/branch-1.6 70b587841 -> fba84d177 [SPARK-14149] Log exceptions in tryOrIOException ## What changes were proposed in this pull request? We ran into a problem today debugging some class loading problem during deserialization, and JVM was masking th

spark git commit: [SPARK-14149] Log exceptions in tryOrIOException

2016-03-25 Thread rxin
Repository: spark Updated Branches: refs/heads/master 20ddf5fdd -> 70a6f0bb5 [SPARK-14149] Log exceptions in tryOrIOException ## What changes were proposed in this pull request? We ran into a problem today debugging some class loading problem during deserialization, and JVM was masking the un

spark git commit: [SPARK-13456][SQL][FOLLOW-UP] lazily generate the outer pointer for case class defined in REPL

2016-03-25 Thread lian
Repository: spark Updated Branches: refs/heads/master 70a6f0bb5 -> e9b6e7d85 [SPARK-13456][SQL][FOLLOW-UP] lazily generate the outer pointer for case class defined in REPL ## What changes were proposed in this pull request? In https://github.com/apache/spark/pull/11410, we missed a corner ca

spark git commit: [SPARK-13887][PYTHON][TRIVIAL][BUILD] Make lint-python script fail fast

2016-03-25 Thread srowen
Repository: spark Updated Branches: refs/heads/master e9b6e7d85 -> 55a605763 [SPARK-13887][PYTHON][TRIVIAL][BUILD] Make lint-python script fail fast ## What changes were proposed in this pull request? Change lint python script to stop on first error rather than building them up so its cleare

spark git commit: [SPARK-13919] [SQL] fix column pruning through filter

2016-03-25 Thread davies
Repository: spark Updated Branches: refs/heads/master 55a605763 -> 6603d9f7e [SPARK-13919] [SQL] fix column pruning through filter ## What changes were proposed in this pull request? This PR fix the conflict between ColumnPruning and PushPredicatesThroughProject, because ColumnPruning will t

spark git commit: [SPARK-14061][SQL] implement CreateMap

2016-03-25 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 6603d9f7e -> 43b15e01c [SPARK-14061][SQL] implement CreateMap ## What changes were proposed in this pull request? As we have `CreateArray` and `CreateStruct`, we should also have `CreateMap`. This PR adds the `CreateMap` expression, and

spark git commit: [SPARK-14144][SQL] Explicitly identify/catch UnsupportedOperationException during parquet reader initialization

2016-03-25 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 43b15e01c -> b5f8c36e3 [SPARK-14144][SQL] Explicitly identify/catch UnsupportedOperationException during parquet reader initialization ## What changes were proposed in this pull request? This PR is a minor cleanup task as part of https:/

spark git commit: [SQL][HOTFIX] Fix flakiness in StateStoreRDDSuite

2016-03-25 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master b5f8c36e3 -> 11fa8741c [SQL][HOTFIX] Fix flakiness in StateStoreRDDSuite ## What changes were proposed in this pull request? StateStoreCoordinator.reportActiveInstance is async, so subsequence state checks must be in eventually. ## How was

spark git commit: [SPARK-12443][SQL] encoderFor should support Decimal

2016-03-25 Thread marmbrus
Repository: spark Updated Branches: refs/heads/master 11fa8741c -> ca003354d [SPARK-12443][SQL] encoderFor should support Decimal ## What changes were proposed in this pull request? JIRA: https://issues.apache.org/jira/browse/SPARK-12443 `constructorFor` will call `dataTypeFor` to determine

spark git commit: [SPARK-14137] [SPARK-14150] [SQL] Infer IsNotNull constraints from non-nullable attributes

2016-03-25 Thread davies
Repository: spark Updated Branches: refs/heads/master ca003354d -> afd0debe0 [SPARK-14137] [SPARK-14150] [SQL] Infer IsNotNull constraints from non-nullable attributes ## What changes were proposed in this pull request? This PR adds support for automatically inferring `IsNotNull` constraints

spark git commit: [SPARK-14131][SQL] Add a workaround for HADOOP-10622 to fix DataFrameReaderWriterSuite

2016-03-25 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master afd0debe0 -> b554b3c46 [SPARK-14131][SQL] Add a workaround for HADOOP-10622 to fix DataFrameReaderWriterSuite ## What changes were proposed in this pull request? There is a potential dead-lock in Hadoop Shell.runCommand before 2.5.0 ([HA

spark git commit: [SPARK-14091][CORE] Improve performance of SparkContext.getCallSite()

2016-03-25 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master b554b3c46 -> ff7cc45f5 [SPARK-14091][CORE] Improve performance of SparkContext.getCallSite() Currently SparkContext.getCallSite() makes a call to Utils.getCallSite(). ``` private[spark] def getCallSite(): CallSite = { val callSite = U

spark git commit: [SPARK-14159][ML] Fixed bug in StringIndexer + related issue in RFormula

2016-03-25 Thread meng
Repository: spark Updated Branches: refs/heads/master ff7cc45f5 -> 54d13bed8 [SPARK-14159][ML] Fixed bug in StringIndexer + related issue in RFormula ## What changes were proposed in this pull request? StringIndexerModel.transform sets the output column metadata to use name inputCol. It sho

[2/2] spark git commit: [SPARK-14073][STREAMING][TEST-MAVEN] Move flume back to Spark

2016-03-25 Thread rxin
[SPARK-14073][STREAMING][TEST-MAVEN] Move flume back to Spark ## What changes were proposed in this pull request? This PR moves flume back to Spark as per the discussion in the dev mail-list. ## How was this patch tested? Existing Jenkins tests. Author: Shixiong Zhu Closes #11895 from zsxwin

[1/2] spark git commit: [SPARK-14073][STREAMING][TEST-MAVEN] Move flume back to Spark

2016-03-25 Thread rxin
Repository: spark Updated Branches: refs/heads/master 54d13bed8 -> 24587ce43 http://git-wip-us.apache.org/repos/asf/spark/blob/24587ce4/external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeInputDStream.scala -- d

spark git commit: [SPARK-14109][SQL] Fix HDFSMetadataLog to fallback from FileContext to FileSystem API

2016-03-25 Thread tdas
Repository: spark Updated Branches: refs/heads/master 24587ce43 -> 13945dd83 [SPARK-14109][SQL] Fix HDFSMetadataLog to fallback from FileContext to FileSystem API ## What changes were proposed in this pull request? HDFSMetadataLog uses newer FileContext API to achieve atomic renaming. Howeve