spark git commit: [SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None

2017-08-09 Thread lixiao
Repository: spark Updated Branches: refs/heads/master c06f3f5ac -> 84454d7d3 [SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None ## What changes were proposed in this pull request? Currently `df.na.replace("*", Map[String, String]("NULL" -> null))` will produce

spark git commit: [SPARK-21551][PYTHON] Increase timeout for PythonRDD.serveIterator

2017-08-09 Thread rxin
Repository: spark Updated Branches: refs/heads/master 0fb73253f -> c06f3f5ac [SPARK-21551][PYTHON] Increase timeout for PythonRDD.serveIterator ## What changes were proposed in this pull request? This modification increases the timeout for `serveIterator` (which is not dynamically

spark git commit: [SPARK-21587][SS] Added filter pushdown through watermarks.

2017-08-09 Thread tdas
Repository: spark Updated Branches: refs/heads/master 2d799d080 -> 0fb73253f [SPARK-21587][SS] Added filter pushdown through watermarks. ## What changes were proposed in this pull request? Push filter predicates through EventTimeWatermark if they're deterministic and do not reference the

spark git commit: [SPARK-21504][SQL] Add spark version info into table metadata

2017-08-09 Thread lixiao
Repository: spark Updated Branches: refs/heads/master b78cf13bf -> 2d799d080 [SPARK-21504][SQL] Add spark version info into table metadata ## What changes were proposed in this pull request? This PR is to add the spark version info in the table metadata. When creating the table, this value

spark git commit: [SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0)

2017-08-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master 83fe3b5e1 -> b78cf13bf [SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0) ## What changes were proposed in this pull request? This pr updated `lz4-java` to the latest (v1.4.0) and removed custom `LZ4BlockInputStream`. We currently

spark git commit: [SPARK-21665][CORE] Need to close resources after use

2017-08-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master 6426adffa -> 83fe3b5e1 [SPARK-21665][CORE] Need to close resources after use ## What changes were proposed in this pull request? Resources in Core - SparkSubmitArguments.scala, Spark-launcher - AbstractCommandBuilder.java,

spark git commit: [SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite

2017-08-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master b35660dd0 -> 6426adffa [SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite Signed-off-by: 10087686 ## What changes were proposed in this pull request? After Unit

spark git commit: [SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite

2017-08-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 f6d56d2f1 -> 3ca55eaaf [SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite Signed-off-by: 10087686 ## What changes were proposed in this pull request? After

spark git commit: [SPARK-21596][SS] Ensure places calling HDFSMetadataLog.get check the return value

2017-08-09 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.2 7446be332 -> f6d56d2f1 [SPARK-21596][SS] Ensure places calling HDFSMetadataLog.get check the return value Same PR as #18799 but for branch 2.2. Main discussion the other PR. When I was investigating a flaky test, I realized

spark git commit: [SPARK-21523][ML] update breeze to 0.13.2 for an emergency bugfix in strong wolfe line search

2017-08-09 Thread yliang
Repository: spark Updated Branches: refs/heads/branch-2.2 d02331452 -> 7446be332 [SPARK-21523][ML] update breeze to 0.13.2 for an emergency bugfix in strong wolfe line search ## What changes were proposed in this pull request? Update breeze to 0.13.1 for an emergency bugfix in strong wolfe

spark git commit: [SPARK-21523][ML] update breeze to 0.13.2 for an emergency bugfix in strong wolfe line search

2017-08-09 Thread yliang
Repository: spark Updated Branches: refs/heads/master ae8a2b149 -> b35660dd0 [SPARK-21523][ML] update breeze to 0.13.2 for an emergency bugfix in strong wolfe line search ## What changes were proposed in this pull request? Update breeze to 0.13.1 for an emergency bugfix in strong wolfe line

spark git commit: [SPARK-21176][WEB UI] Use a single ProxyServlet to proxy all workers and applications

2017-08-09 Thread wenchen
Repository: spark Updated Branches: refs/heads/master f016f5c8f -> ae8a2b149 [SPARK-21176][WEB UI] Use a single ProxyServlet to proxy all workers and applications ## What changes were proposed in this pull request? Currently, each application and each worker creates their own proxy servlet.