spark git commit: [SPARK-20848][SQL][FOLLOW-UP] Shutdown the pool after reading parquet files

2017-05-24 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.1 c3302e81e -> 7015f6f0e [SPARK-20848][SQL][FOLLOW-UP] Shutdown the pool after reading parquet files ## What changes were proposed in this pull request? This is a follow-up to #18073. Taking a safer approach to shutdown the pool to

spark git commit: [SPARK-20848][SQL][FOLLOW-UP] Shutdown the pool after reading parquet files

2017-05-24 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 3f82d65bf -> e0aa23939 [SPARK-20848][SQL][FOLLOW-UP] Shutdown the pool after reading parquet files ## What changes were proposed in this pull request? This is a follow-up to #18073. Taking a safer approach to shutdown the pool to

spark git commit: [SPARK-20848][SQL][FOLLOW-UP] Shutdown the pool after reading parquet files

2017-05-24 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 197f9018a -> 6b68d61cf [SPARK-20848][SQL][FOLLOW-UP] Shutdown the pool after reading parquet files ## What changes were proposed in this pull request? This is a follow-up to #18073. Taking a safer approach to shutdown the pool to prevent

spark git commit: [SPARK-20403][SQL] Modify the instructions of some functions

2017-05-24 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.2 ae65d3014 -> 3f82d65bf [SPARK-20403][SQL] Modify the instructions of some functions ## What changes were proposed in this pull request? 1.add instructions of 'cast' function When using 'show functions' and 'desc function cast'

spark git commit: [SPARK-20403][SQL] Modify the instructions of some functions

2017-05-24 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 5f8ff2fc9 -> 197f9018a [SPARK-20403][SQL] Modify the instructions of some functions ## What changes were proposed in this pull request? 1.add instructions of 'cast' function When using 'show functions' and 'desc function cast'

spark git commit: [SPARK-18406][CORE][BACKPORT-2.1] Race between end-of-task and completion iterator read lock release

2017-05-24 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.1 2f68631f5 -> c3302e81e [SPARK-18406][CORE][BACKPORT-2.1] Race between end-of-task and completion iterator read lock release This is a backport PR of #18076 to 2.1. ## What changes were proposed in this pull request? When a

spark git commit: [SPARK-16202][SQL][DOC] Follow-up to Correct The Description of CreatableRelationProvider's createRelation

2017-05-24 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.2 2405afce4 -> ae65d3014 [SPARK-16202][SQL][DOC] Follow-up to Correct The Description of CreatableRelationProvider's createRelation ## What changes were proposed in this pull request? Follow-up to SPARK-16202: 1. Remove the

spark git commit: [SPARK-16202][SQL][DOC] Follow-up to Correct The Description of CreatableRelationProvider's createRelation

2017-05-24 Thread lixiao
Repository: spark Updated Branches: refs/heads/master c0b3e45e3 -> 5f8ff2fc9 [SPARK-16202][SQL][DOC] Follow-up to Correct The Description of CreatableRelationProvider's createRelation ## What changes were proposed in this pull request? Follow-up to SPARK-16202: 1. Remove the duplication of

spark git commit: [SPARK-20872][SQL] ShuffleExchange.nodeName should handle null coordinator

2017-05-24 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.2 b7a2a16b1 -> 2405afce4 [SPARK-20872][SQL] ShuffleExchange.nodeName should handle null coordinator ## What changes were proposed in this pull request? A one-liner change in `ShuffleExchange.nodeName` to cover the case when

spark git commit: [SPARK-20872][SQL] ShuffleExchange.nodeName should handle null coordinator

2017-05-24 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 95aef660b -> c0b3e45e3 [SPARK-20872][SQL] ShuffleExchange.nodeName should handle null coordinator ## What changes were proposed in this pull request? A one-liner change in `ShuffleExchange.nodeName` to cover the case when `coordinator`

spark git commit: [SPARK-20205][CORE] Make sure StageInfo is updated before sending event.

2017-05-24 Thread vanzin
Repository: spark Updated Branches: refs/heads/master a64746677 -> 95aef660b [SPARK-20205][CORE] Make sure StageInfo is updated before sending event. The DAGScheduler was sending a "stage submitted" event before it properly updated the event's information. This meant that a listener (e.g. the

spark git commit: [SPARK-18406][CORE][BACKPORT-2.0] Race between end-of-task and completion iterator read lock release

2017-05-24 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.0 72e1f83d7 -> 79fbfbbc7 [SPARK-18406][CORE][BACKPORT-2.0] Race between end-of-task and completion iterator read lock release This is a backport PR of #18076 to 2.0 and 2.1. ## What changes were proposed in this pull request? When a

spark git commit: [SPARK-20867][SQL] Move hints from Statistics into HintInfo class

2017-05-24 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.2 c59ad420b -> b7a2a16b1 [SPARK-20867][SQL] Move hints from Statistics into HintInfo class ## What changes were proposed in this pull request? This is a follow-up to SPARK-20857 to move the broadcast hint from Statistics into a new

spark git commit: [SPARK-20867][SQL] Move hints from Statistics into HintInfo class

2017-05-24 Thread lixiao
Repository: spark Updated Branches: refs/heads/master f72ad303f -> a64746677 [SPARK-20867][SQL] Move hints from Statistics into HintInfo class ## What changes were proposed in this pull request? This is a follow-up to SPARK-20857 to move the broadcast hint from Statistics into a new HintInfo

spark git commit: [SPARK-20848][SQL] Shutdown the pool after reading parquet files

2017-05-24 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.1 13adc0fc0 -> 2f68631f5 [SPARK-20848][SQL] Shutdown the pool after reading parquet files ## What changes were proposed in this pull request? >From JIRA: On each call to spark.read.parquet, a new ForkJoinPool is created. >One of the

spark git commit: [SPARK-20848][SQL] Shutdown the pool after reading parquet files

2017-05-24 Thread wenchen
Repository: spark Updated Branches: refs/heads/master bc66a77bb -> f72ad303f [SPARK-20848][SQL] Shutdown the pool after reading parquet files ## What changes were proposed in this pull request? >From JIRA: On each call to spark.read.parquet, a new ForkJoinPool is created. >One of the

spark git commit: [SPARK-20862][MLLIB][PYTHON] Avoid passing float to ndarray.reshape in LogisticRegressionModel

2017-05-24 Thread yliang
Repository: spark Updated Branches: refs/heads/branch-2.0 4dd34d004 -> 72e1f83d7 [SPARK-20862][MLLIB][PYTHON] Avoid passing float to ndarray.reshape in LogisticRegressionModel ## What changes were proposed in this pull request? Fixed TypeError with python3 and numpy 1.12.1. Numpy's

spark git commit: [SPARK-20862][MLLIB][PYTHON] Avoid passing float to ndarray.reshape in LogisticRegressionModel

2017-05-24 Thread yliang
Repository: spark Updated Branches: refs/heads/branch-2.1 f4538c95f -> 13adc0fc0 [SPARK-20862][MLLIB][PYTHON] Avoid passing float to ndarray.reshape in LogisticRegressionModel ## What changes were proposed in this pull request? Fixed TypeError with python3 and numpy 1.12.1. Numpy's

spark git commit: [SPARK-20862][MLLIB][PYTHON] Avoid passing float to ndarray.reshape in LogisticRegressionModel

2017-05-24 Thread yliang
Repository: spark Updated Branches: refs/heads/branch-2.2 1d107242f -> 83aeac9e0 [SPARK-20862][MLLIB][PYTHON] Avoid passing float to ndarray.reshape in LogisticRegressionModel ## What changes were proposed in this pull request? Fixed TypeError with python3 and numpy 1.12.1. Numpy's

spark git commit: [SPARK-20862][MLLIB][PYTHON] Avoid passing float to ndarray.reshape in LogisticRegressionModel

2017-05-24 Thread yliang
Repository: spark Updated Branches: refs/heads/master 1816eb3be -> bc66a77bb [SPARK-20862][MLLIB][PYTHON] Avoid passing float to ndarray.reshape in LogisticRegressionModel ## What changes were proposed in this pull request? Fixed TypeError with python3 and numpy 1.12.1. Numpy's `reshape` no

spark git commit: [SPARK-20631][FOLLOW-UP] Fix incorrect tests.

2017-05-24 Thread yliang
Repository: spark Updated Branches: refs/heads/branch-2.2 e936a96ba -> 1d107242f [SPARK-20631][FOLLOW-UP] Fix incorrect tests. ## What changes were proposed in this pull request? - Fix incorrect tests for `_check_thresholds`. - Move test to `ParamTests`. ## How was this patch tested? Unit

spark git commit: [SPARK-20631][FOLLOW-UP] Fix incorrect tests.

2017-05-24 Thread yliang
Repository: spark Updated Branches: refs/heads/master 9afcf127d -> 1816eb3be [SPARK-20631][FOLLOW-UP] Fix incorrect tests. ## What changes were proposed in this pull request? - Fix incorrect tests for `_check_thresholds`. - Move test to `ParamTests`. ## How was this patch tested? Unit

spark git commit: [SPARK-20764][ML][PYSPARK][FOLLOWUP] Fix visibility discrepancy with numInstances and degreesOfFreedom in LR and GLR - Python version

2017-05-24 Thread yliang
Repository: spark Updated Branches: refs/heads/branch-2.2 ee9d5975e -> e936a96ba [SPARK-20764][ML][PYSPARK][FOLLOWUP] Fix visibility discrepancy with numInstances and degreesOfFreedom in LR and GLR - Python version ## What changes were proposed in this pull request? Add test cases for

spark git commit: [SPARK-20764][ML][PYSPARK][FOLLOWUP] Fix visibility discrepancy with numInstances and degreesOfFreedom in LR and GLR - Python version

2017-05-24 Thread yliang
Repository: spark Updated Branches: refs/heads/master d76633e3c -> 9afcf127d [SPARK-20764][ML][PYSPARK][FOLLOWUP] Fix visibility discrepancy with numInstances and degreesOfFreedom in LR and GLR - Python version ## What changes were proposed in this pull request? Add test cases for PR-18062

spark git commit: [SPARK-18406][CORE] Race between end-of-task and completion iterator read lock release

2017-05-24 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 9434280cf -> d76633e3c [SPARK-18406][CORE] Race between end-of-task and completion iterator read lock release ## What changes were proposed in this pull request? When a TaskContext is not propagated properly to all child threads for the

spark git commit: [SPARK-18406][CORE] Race between end-of-task and completion iterator read lock release

2017-05-24 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 00dee3902 -> ee9d5975e [SPARK-18406][CORE] Race between end-of-task and completion iterator read lock release ## What changes were proposed in this pull request? When a TaskContext is not propagated properly to all child threads for