spark git commit: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8String Serializable

2017-07-17 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 83bdb0487 -> 99ce551a1 [SPARK-21445] Make IntWrapper and LongWrapper in UTF8String Serializable ## What changes were proposed in this pull request? Making those two classes will avoid Serialization issues like below: ``` Caused by:

spark git commit: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8String Serializable

2017-07-17 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 0be5fb41a -> 26cd2ca04 [SPARK-21445] Make IntWrapper and LongWrapper in UTF8String Serializable ## What changes were proposed in this pull request? Making those two classes will avoid Serialization issues like below: ``` Caused by:

spark git commit: [SPARK-21332][SQL] Incorrect result type inferred for some decimal expressions

2017-07-17 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.0 e4f57f2b3 -> d7b9d6235 [SPARK-21332][SQL] Incorrect result type inferred for some decimal expressions ## What changes were proposed in this pull request? This PR changes the direction of expression transformation in the

spark git commit: [SPARK-21332][SQL] Incorrect result type inferred for some decimal expressions

2017-07-17 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.1 a9efce46b -> caf32b3ca [SPARK-21332][SQL] Incorrect result type inferred for some decimal expressions ## What changes were proposed in this pull request? This PR changes the direction of expression transformation in the

spark git commit: [SPARK-21332][SQL] Incorrect result type inferred for some decimal expressions

2017-07-17 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 5952ad2b4 -> 0be5fb41a [SPARK-21332][SQL] Incorrect result type inferred for some decimal expressions ## What changes were proposed in this pull request? This PR changes the direction of expression transformation in the DecimalPrecision

spark git commit: [SPARK-21332][SQL] Incorrect result type inferred for some decimal expressions

2017-07-17 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.2 0ef98fd43 -> 83bdb0487 [SPARK-21332][SQL] Incorrect result type inferred for some decimal expressions ## What changes were proposed in this pull request? This PR changes the direction of expression transformation in the

spark git commit: [SPARK-21444] Be more defensive when removing broadcasts in MapOutputTracker

2017-07-17 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master e9faae135 -> 5952ad2b4 [SPARK-21444] Be more defensive when removing broadcasts in MapOutputTracker ## What changes were proposed in this pull request? In SPARK-21444, sitalkedia reported an issue where the `Broadcast.destroy()` call in

spark git commit: [SPARK-21409][SS] Follow up PR to allow different types of custom metrics to be exposed

2017-07-17 Thread tdas
Repository: spark Updated Branches: refs/heads/master 7aac755ba -> e9faae135 [SPARK-21409][SS] Follow up PR to allow different types of custom metrics to be exposed ## What changes were proposed in this pull request? Implementation may expose both timing as well as size metrics. This PR

spark git commit: [SPARK-21410][CORE] Create less partitions for RangePartitioner if RDD.count() is less than `partitions`

2017-07-17 Thread wenchen
Repository: spark Updated Branches: refs/heads/master a8c6d0f64 -> 7aac755ba [SPARK-21410][CORE] Create less partitions for RangePartitioner if RDD.count() is less than `partitions` ## What changes were proposed in this pull request? Fix a bug in RangePartitioner: In

spark git commit: [SPARK-19104][BACKPORT-2.1][SQL] Lambda variables in ExternalMapToCatalyst should be global

2017-07-17 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.1 ca4d2aa39 -> a9efce46b [SPARK-19104][BACKPORT-2.1][SQL] Lambda variables in ExternalMapToCatalyst should be global ## What changes were proposed in this pull request? This PR is backport of #18418 to Spark 2.1.

spark git commit: [MINOR] Improve SQLConf messages

2017-07-17 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 9d8c83179 -> a8c6d0f64 [MINOR] Improve SQLConf messages ### What changes were proposed in this pull request? The current SQLConf messages of `spark.sql.hive.convertMetastoreParquet` and `spark.sql.hive.convertMetastoreOrc` are not very

spark git commit: [SPARK-21409][SS] Expose state store memory usage in SQL metrics and progress updates

2017-07-17 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 53465075c -> 9d8c83179 [SPARK-21409][SS] Expose state store memory usage in SQL metrics and progress updates ## What changes were proposed in this pull request? Currently, there is no tracking of memory usage of state stores. This JIRA

spark git commit: [SPARK-21321][SPARK CORE] Spark very verbose on shutdown

2017-07-17 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-2.2 8e85ce625 -> 0ef98fd43 [SPARK-21321][SPARK CORE] Spark very verbose on shutdown ## What changes were proposed in this pull request? The current code is very verbose on shutdown. The changes I propose is to change the log level when

spark git commit: [SPARK-21321][SPARK CORE] Spark very verbose on shutdown

2017-07-17 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 7047f49f4 -> 0e07a29cf [SPARK-21321][SPARK CORE] Spark very verbose on shutdown ## What changes were proposed in this pull request? The current code is very verbose on shutdown. The changes I propose is to change the log level when the

spark git commit: [SPARK-21221][ML] CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest

2017-07-17 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 4ce735eed -> 7047f49f4 [SPARK-21221][ML] CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest ## What changes were proposed in this pull request? Added functionality for CrossValidator and

spark git commit: [SPARK-21394][SPARK-21432][PYTHON] Reviving callable object/partial function support in UDF in PySpark

2017-07-17 Thread holden
Repository: spark Updated Branches: refs/heads/master e398c2814 -> 4ce735eed [SPARK-21394][SPARK-21432][PYTHON] Reviving callable object/partial function support in UDF in PySpark ## What changes were proposed in this pull request? This PR proposes to avoid `__name__` in the tuple naming

spark git commit: [SPARK-21354][SQL] INPUT FILE related functions do not support more than one sources

2017-07-17 Thread wenchen
Repository: spark Updated Branches: refs/heads/master fd52a747f -> e398c2814 [SPARK-21354][SQL] INPUT FILE related functions do not support more than one sources ### What changes were proposed in this pull request? The build-in functions `input_file_name`, `input_file_block_start`,