spark git commit: [HOT-FIX][BUILD] Use the new location of `checkstyle-suppressions.xml`

Repository: spark Updated Branches: refs/heads/master e52e597db -> 7771c7314 [HOT-FIX][BUILD] Use the new location of `checkstyle-suppressions.xml` ## What changes were proposed in this pull request? This PR fixes `dev/lint-java` and `mvn checkstyle:check` failures due the recent file locati

spark git commit: [SPARK-13117][WEB UI] WebUI should use the local ip not 0.0.0.0

Repository: spark Updated Branches: refs/heads/master 7771c7314 -> 9bf76ddde [SPARK-13117][WEB UI] WebUI should use the local ip not 0.0.0.0 ## What changes were proposed in this pull request? In WebUI, now Jetty Server starts with SPARK_LOCAL_IP config value if it is configured otherwise it

spark git commit: [SPARK-13675][UI] Fix wrong historyserver url link for application running in yarn cluster mode

Repository: spark Updated Branches: refs/heads/master 9bf76ddde -> 9e86e6efd [SPARK-13675][UI] Fix wrong historyserver url link for application running in yarn cluster mode ## What changes were proposed in this pull request? Current URL for each application to access history UI is like: http

spark git commit: [SPARK-13637][SQL] use more information to simplify the code in Expand builder

Repository: spark Updated Branches: refs/heads/master 9e86e6efd -> 7d05d02bf [SPARK-13637][SQL] use more information to simplify the code in Expand builder ## What changes were proposed in this pull request? The code in `Expand.apply` can be simplified by existing information: * the `groupBy

spark git commit: [SPARK-13715][MLLIB] Remove last usages of jblas in tests

Repository: spark Updated Branches: refs/heads/master ca1a7b9d6 -> 54040f8d3 [SPARK-13715][MLLIB] Remove last usages of jblas in tests ## What changes were proposed in this pull request? Remove last usage of jblas, in tests ## How was this patch tested? Jenkins tests -- the same ones that a

spark git commit: [HOTFIX][YARN] Fix yarn cluster mode fire and forget regression

Repository: spark Updated Branches: refs/heads/master 7d05d02bf -> ca1a7b9d6 [HOTFIX][YARN] Fix yarn cluster mode fire and forget regression ## What changes were proposed in this pull request? Fire and forget is disabled by default, with this patch #10205 it is enabled by default, so this is

spark git commit: [SPARK-13657] [SQL] Support parsing very long AND/OR expressions

Repository: spark Updated Branches: refs/heads/master 54040f8d3 -> 78d3b6051 [SPARK-13657] [SQL] Support parsing very long AND/OR expressions ## What changes were proposed in this pull request? In order to avoid StackOverflow when parse a expression with hundreds of ORs, we should use loop i

spark git commit: [SPARK-13695] Don't cache MEMORY_AND_DISK blocks as bytes in memory after spills

Repository: spark Updated Branches: refs/heads/master 78d3b6051 -> ad3c9a973 [SPARK-13695] Don't cache MEMORY_AND_DISK blocks as bytes in memory after spills When a cached block is spilled to disk and read back in serialized form (i.e. as bytes), the current BlockManager implementation will a

spark git commit: [SPARK-12727][SQL] support SQL generation for aggregate with multi-distinct

Repository: spark Updated Branches: refs/heads/master ad3c9a973 -> 46881b4ea [SPARK-12727][SQL] support SQL generation for aggregate with multi-distinct ## What changes were proposed in this pull request? This PR add SQL generation support for aggregate with multi-distinct, by simply moving

spark git commit: [ML] testEstimatorAndModelReadWrite should call checkModelData

Repository: spark Updated Branches: refs/heads/master 46881b4ea -> 9740954f3 [ML] testEstimatorAndModelReadWrite should call checkModelData ## What changes were proposed in this pull request? Although we defined ```checkModelData``` in [```read/write``` test](https://github.com/apache/spark/b

spark git commit: [SPARK-13740][SQL] add null check for _verify_type in types.py

Repository: spark Updated Branches: refs/heads/master 9740954f3 -> d5ce61722 [SPARK-13740][SQL] add null check for _verify_type in types.py ## What changes were proposed in this pull request? This PR adds null check in `_verify_type` according to the nullability information. ## How was this

spark git commit: [SPARK-13593] [SQL] improve the `createDataFrame` to accept data type string and verify the data

Repository: spark Updated Branches: refs/heads/master d5ce61722 -> d57daf1f7 [SPARK-13593] [SQL] improve the `createDataFrame` to accept data type string and verify the data ## What changes were proposed in this pull request? This PR improves the `createDataFrame` method to make it also acce

spark git commit: [SPARK-13400] Stop using deprecated Octal escape literals

Repository: spark Updated Branches: refs/heads/master d57daf1f7 -> 076009b94 [SPARK-13400] Stop using deprecated Octal escape literals ## What changes were proposed in this pull request? This removes the remaining deprecated Octal escape literals. The followings are the warnings on those two

spark git commit: [SPARK-13738][SQL] Cleanup Data Source resolution

Repository: spark Updated Branches: refs/heads/master 076009b94 -> 1e2884059 [SPARK-13738][SQL] Cleanup Data Source resolution Follow-up to #11509, that simply refactors the interface that we use when resolving a pluggable `DataSource`. - Multiple functions share the same set of arguments so

spark git commit: [SPARK-13668][SQL] Reorder filter/join predicates to short-circuit isNotNull checks

Repository: spark Updated Branches: refs/heads/master 1e2884059 -> e430614ea [SPARK-13668][SQL] Reorder filter/join predicates to short-circuit isNotNull checks ## What changes were proposed in this pull request? If a filter predicate or a join condition consists of `IsNotNull` checks, we s

spark git commit: [SPARK-13755] Escape quotes in SQL plan visualization node labels

Repository: spark Updated Branches: refs/heads/master e430614ea -> 81f54acc9 [SPARK-13755] Escape quotes in SQL plan visualization node labels When generating Graphviz DOT files in the SQL query visualization we need to escape double-quotes inside node labels. This is a followup to #11309, wh

spark git commit: [SPARK-13755] Escape quotes in SQL plan visualization node labels

Repository: spark Updated Branches: refs/heads/branch-1.6 bace137f2 -> 8ec4f159a [SPARK-13755] Escape quotes in SQL plan visualization node labels When generating Graphviz DOT files in the SQL query visualization we need to escape double-quotes inside node labels. This is a followup to #11309

spark git commit: [SPARK-13625][PYSPARK][ML] Added a check to see if an attribute is a property when getting param list

Repository: spark Updated Branches: refs/heads/master 81f54acc9 -> d8813fa04 [SPARK-13625][PYSPARK][ML] Added a check to see if an attribute is a property when getting param list ## What changes were proposed in this pull request? Added a check in pyspark.ml.param.Param.params() to see if an

spark git commit: [SPARK-13750][SQL] fix sizeInBytes of HadoopFsRelation

Repository: spark Updated Branches: refs/heads/master d8813fa04 -> 982ef2b87 [SPARK-13750][SQL] fix sizeInBytes of HadoopFsRelation ## What changes were proposed in this pull request? This PR fix the sizeInBytes of HadoopFsRelation. ## How was this patch tested? Added regression test for th

spark git commit: [SPARK-13754] Keep old data source name for backwards compatibility

Repository: spark Updated Branches: refs/heads/master 982ef2b87 -> cc4ab37ee [SPARK-13754] Keep old data source name for backwards compatibility ## Motivation CSV data source was contributed by Databricks. It is the inlined version of https://github.com/databricks/spark-csv. The data source n

spark git commit: [SPARK-7286][SQL] Deprecate !== in favour of =!=

Repository: spark Updated Branches: refs/heads/master cc4ab37ee -> 035d3acdf [SPARK-7286][SQL] Deprecate !== in favour of =!= This PR replaces #9925 which had issues with CI. **Please see the original PR for any previous discussions.** ## What changes were proposed in this pull request? Depr