[1/2] spark git commit: [SPARK-18278][SCHEDULER] Spark on Kubernetes - Basic Scheduler Backend

2017-11-28 Thread rxin
Repository: spark Updated Branches: refs/heads/master 475a29f11 -> e9b2070ab http://git-wip-us.apache.org/repos/asf/spark/blob/e9b2070a/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala

[2/2] spark git commit: [SPARK-18278][SCHEDULER] Spark on Kubernetes - Basic Scheduler Backend

2017-11-28 Thread rxin
[SPARK-18278][SCHEDULER] Spark on Kubernetes - Basic Scheduler Backend ## What changes were proposed in this pull request? This is a stripped down version of the `KubernetesClusterSchedulerBackend` for Spark with the following components: - Static Allocation of Executors - Executor Pod Factory

spark git commit: [SPARK-22637][SQL] Only refresh a logical plan once.

2017-11-28 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.2 eef72d3f0 -> 38a0532cf [SPARK-22637][SQL] Only refresh a logical plan once. ## What changes were proposed in this pull request? `CatalogImpl.refreshTable` uses `foreach(..)` to refresh all tables in a view. This traverses all nodes in

spark git commit: [SPARK-22637][SQL] Only refresh a logical plan once.

2017-11-28 Thread lixiao
Repository: spark Updated Branches: refs/heads/master a10b328db -> 475a29f11 [SPARK-22637][SQL] Only refresh a logical plan once. ## What changes were proposed in this pull request? `CatalogImpl.refreshTable` uses `foreach(..)` to refresh all tables in a view. This traverses all nodes in the

spark git commit: [SPARK-22431][SQL] Ensure that the datatype in the schema for the table/view metadata is parseable by Spark before persisting it

2017-11-28 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master da3557429 -> a10b328db [SPARK-22431][SQL] Ensure that the datatype in the schema for the table/view metadata is parseable by Spark before persisting it ## What changes were proposed in this pull request? * JIRA:

spark git commit: [SPARK-22515][SQL] Estimation relation size based on numRows * rowSize

2017-11-28 Thread lixiao
Repository: spark Updated Branches: refs/heads/master b70e483cb -> da3557429 [SPARK-22515][SQL] Estimation relation size based on numRows * rowSize ## What changes were proposed in this pull request? Currently, relation size is computed as the sum of file size, which is error-prone because

spark git commit: [SPARK-22617][SQL] make splitExpressions extract current input of the context

2017-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 1e07fff24 -> b70e483cb [SPARK-22617][SQL] make splitExpressions extract current input of the context ## What changes were proposed in this pull request? Mostly when we call `CodegenContext.splitExpressions`, we want to split the code

spark git commit: [SPARK-22520][SQL][FOLLOWUP] remove outer if for case when codegen

2017-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 64817c423 -> 1e07fff24 [SPARK-22520][SQL][FOLLOWUP] remove outer if for case when codegen ## What changes were proposed in this pull request? a minor cleanup for https://github.com/apache/spark/pull/19752 . Remove the outer if as the

spark git commit: [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp values for Pandas to respect session timezone

2017-11-28 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 33d43bf1b -> 64817c423 [SPARK-22395][SQL][PYTHON] Fix the behavior of timestamp values for Pandas to respect session timezone ## What changes were proposed in this pull request? When converting Pandas DataFrame/Series from/to Spark