spark git commit: [SPARK-19753][CORE] Un-register all shuffle output on a host in case of slave lost or fetch failure

2017-06-13 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 8b5b2e272 -> dccc0aa3c [SPARK-19753][CORE] Un-register all shuffle output on a host in case of slave lost or fetch failure ## What changes were proposed in this pull request? Currently, when we detect fetch failure, we only remove the

spark git commit: [SPARK-20986][SQL] Reset table's statistics after PruneFileSourcePartitions rule.

2017-06-13 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 53212c32b -> 42cc83082 [SPARK-20986][SQL] Reset table's statistics after PruneFileSourcePartitions rule. ## What changes were proposed in this pull request? After PruneFileSourcePartitions rule, It needs reset table's statistics

spark git commit: [SPARK-20986][SQL] Reset table's statistics after PruneFileSourcePartitions rule.

2017-06-13 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 9eb095243 -> 8b5b2e272 [SPARK-20986][SQL] Reset table's statistics after PruneFileSourcePartitions rule. ## What changes were proposed in this pull request? After PruneFileSourcePartitions rule, It needs reset table's statistics because

spark git commit: [SPARK-12552][CORE] Correctly count the driver resource when recovering from failure for Master

2017-06-13 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 7ba8bf288 -> 9eb095243 [SPARK-12552][CORE] Correctly count the driver resource when recovering from failure for Master Currently in Standalone HA mode, the resource usage of driver is not correctly counted in Master when recovering from

spark git commit: [SPARK-12552][CORE] Correctly count the driver resource when recovering from failure for Master

2017-06-13 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 220943d85 -> 53212c32b [SPARK-12552][CORE] Correctly count the driver resource when recovering from failure for Master Currently in Standalone HA mode, the resource usage of driver is not correctly counted in Master when recovering

spark git commit: [SPARK-20979][SS] Add RateSource to generate values for tests and benchmark

2017-06-13 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-2.2 2bc2c1539 -> 220943d85 [SPARK-20979][SS] Add RateSource to generate values for tests and benchmark ## What changes were proposed in this pull request? This PR adds RateSource for Structured Streaming so that the user can use it to

spark git commit: [SPARK-21016][CORE] Improve code fault tolerance for converting string to number

2017-06-13 Thread lixiao
Repository: spark Updated Branches: refs/heads/master bcf3643f9 -> 7ba8bf288 [SPARK-21016][CORE] Improve code fault tolerance for converting string to number ## What changes were proposed in this pull request? When converting `string` to `number`(int, long or double), if the string has a

spark git commit: [SPARK-21051][SQL] Add hash map metrics to aggregate

2017-06-13 Thread lixiao
Repository: spark Updated Branches: refs/heads/master b36ce2a24 -> bcf3643f9 [SPARK-21051][SQL] Add hash map metrics to aggregate ## What changes were proposed in this pull request? This adds the average hash map probe metrics to hash aggregate. `BytesToBytesMap` already has API to get the

spark git commit: [SPARK-21064][CORE][TEST] Fix the default value bug in NettyBlockTransferServiceSuite

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 58a8a379d -> ee0e74e65 [SPARK-21064][CORE][TEST] Fix the default value bug in NettyBlockTransferServiceSuite ## What changes were proposed in this pull request? The default value for `spark.port.maxRetries` is 100, but we use 10 in

spark git commit: [SPARK-21064][CORE][TEST] Fix the default value bug in NettyBlockTransferServiceSuite

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 039c46506 -> 2bc2c1539 [SPARK-21064][CORE][TEST] Fix the default value bug in NettyBlockTransferServiceSuite ## What changes were proposed in this pull request? The default value for `spark.port.maxRetries` is 100, but we use 10 in

spark git commit: [SPARK-21064][CORE][TEST] Fix the default value bug in NettyBlockTransferServiceSuite

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master b7304f255 -> b36ce2a24 [SPARK-21064][CORE][TEST] Fix the default value bug in NettyBlockTransferServiceSuite ## What changes were proposed in this pull request? The default value for `spark.port.maxRetries` is 100, but we use 10 in the

spark git commit: [SPARK-21060][WEB-UI] Css style about paging function is error in the executor page. Css style about paging function is error in the executor page. It is different of history server

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 9b2c877be -> b7304f255 [SPARK-21060][WEB-UI] Css style about paging function is error in the executor page. Css style about paging function is error in the executor page. It is different of history server ui paging function css style. ##

spark git commit: [SPARK-21060][WEB-UI] Css style about paging function is error in the executor page. Css style about paging function is error in the executor page. It is different of history server

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 24836be54 -> 039c46506 [SPARK-21060][WEB-UI] Css style about paging function is error in the executor page. Css style about paging function is error in the executor page. It is different of history server ui paging function css style.

spark git commit: [SPARK-21039][SPARK CORE] Use treeAggregate instead of aggregate in DataFrame.stat.bloomFilter

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2aaed0a4d -> 9b2c877be [SPARK-21039][SPARK CORE] Use treeAggregate instead of aggregate in DataFrame.stat.bloomFilter ## What changes were proposed in this pull request? To use treeAggregate instead of aggregate in

spark git commit: [SPARK-21006][TESTS][FOLLOW-UP] Some Worker's RpcEnv is leaked in WorkerSuite

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 7b7c85ede -> 2aaed0a4d [SPARK-21006][TESTS][FOLLOW-UP] Some Worker's RpcEnv is leaked in WorkerSuite ## What changes were proposed in this pull request? Create rpcEnv and run later needs shutdown. as #18226 ## How was this patch tested?

spark git commit: [SPARK-20920][SQL] ForkJoinPool pools are leaked when writing hive tables with many partitions

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 03cc18ba1 -> 58a8a379d [SPARK-20920][SQL] ForkJoinPool pools are leaked when writing hive tables with many partitions ## What changes were proposed in this pull request? Don't leave thread pool running from

spark git commit: [SPARK-20920][SQL] ForkJoinPool pools are leaked when writing hive tables with many partitions

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 278ba7a2c -> 7b7c85ede [SPARK-20920][SQL] ForkJoinPool pools are leaked when writing hive tables with many partitions ## What changes were proposed in this pull request? Don't leave thread pool running from

spark git commit: [SPARK-20920][SQL] ForkJoinPool pools are leaked when writing hive tables with many partitions

2017-06-13 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 dae1a9875 -> 24836be54 [SPARK-20920][SQL] ForkJoinPool pools are leaked when writing hive tables with many partitions ## What changes were proposed in this pull request? Don't leave thread pool running from