spark git commit: [SPARK-4446] [SPARK CORE]

Repository: spark Updated Branches: refs/heads/master 0eb4a7fb0 -> e216ffaea [SPARK-4446] [SPARK CORE] MetadataCleaner schedule task with a wrong param for delay time . Author: Leolh Closes #3306 from Leolh/master and squashes the following commits: 4a21f4e [Leolh] Update MetadataCleaner.s

spark git commit: [SPARK-4446] [SPARK CORE]

Repository: spark Updated Branches: refs/heads/branch-1.2 4a5c3d21b -> 83d24efb0 [SPARK-4446] [SPARK CORE] MetadataCleaner schedule task with a wrong param for delay time . Author: Leolh Closes #3306 from Leolh/master and squashes the following commits: 4a21f4e [Leolh] Update MetadataClean

spark git commit: [SPARK-4480] Avoid many small spills in external data structures

Repository: spark Updated Branches: refs/heads/master 73fedf5a6 -> 0eb4a7fb0 [SPARK-4480] Avoid many small spills in external data structures **Summary.** Currently, we may spill many small files in `ExternalAppendOnlyMap` and `ExternalSorter`. The underlying root cause of this is summarized

spark git commit: [SPARK-4480] Avoid many small spills in external data structures

Repository: spark Updated Branches: refs/heads/branch-1.2 f21e550e3 -> 4a5c3d21b [SPARK-4480] Avoid many small spills in external data structures **Summary.** Currently, we may spill many small files in `ExternalAppendOnlyMap` and `ExternalSorter`. The underlying root cause of this is summar

spark git commit: [Spark-4484] Treat maxResultSize as unlimited when set to 0; improve error message

Repository: spark Updated Branches: refs/heads/master 9ccc53c72 -> 73fedf5a6 [Spark-4484] Treat maxResultSize as unlimited when set to 0; improve error message The check for maxResultSize > 0 is missing, results in failures. Also, error message needs to be improved so the developers know tha

spark git commit: [Spark-4484] Treat maxResultSize as unlimited when set to 0; improve error message

Repository: spark Updated Branches: refs/heads/branch-1.2 d68b40bfc -> f21e550e3 [Spark-4484] Treat maxResultSize as unlimited when set to 0; improve error message The check for maxResultSize > 0 is missing, results in failures. Also, error message needs to be improved so the developers know

spark git commit: [SPARK-4478] Keep totalRegisteredExecutors up-to-date

Repository: spark Updated Branches: refs/heads/branch-1.2 8786ddd48 -> d68b40bfc [SPARK-4478] Keep totalRegisteredExecutors up-to-date This rebases PR 3368. This commit fixes totalRegisteredExecutors update [SPARK-4478], so that we can correctly keep track of number of registered executors.

spark git commit: [SPARK-4478] Keep totalRegisteredExecutors up-to-date

Repository: spark Updated Branches: refs/heads/master 377b06820 -> 9ccc53c72 [SPARK-4478] Keep totalRegisteredExecutors up-to-date This rebases PR 3368. This commit fixes totalRegisteredExecutors update [SPARK-4478], so that we can correctly keep track of number of registered executors. Aut

spark git commit: Updating GraphX programming guide and documentation

Repository: spark Updated Branches: refs/heads/master 04d462f64 -> 377b06820 Updating GraphX programming guide and documentation This pull request revises the programming guide to reflect changes in the GraphX API as well as the deprecated mapReduceTriplets operator. Author: Joseph E. Gonzal

spark git commit: Updating GraphX programming guide and documentation

Repository: spark Updated Branches: refs/heads/branch-1.2 a7c64cc8f -> 8786ddd48 Updating GraphX programming guide and documentation This pull request revises the programming guide to reflect changes in the GraphX API as well as the deprecated mapReduceTriplets operator. Author: Joseph E. Go

spark git commit: [SPARK-4495] Fix memory leak in JobProgressListener

Repository: spark Updated Branches: refs/heads/branch-1.2 a250ca369 -> a7c64cc8f [SPARK-4495] Fix memory leak in JobProgressListener This commit fixes a memory leak in JobProgressListener that I introduced in SPARK-2321 and adds a testing framework to ensure that it’s very difficult to ina

spark git commit: [SPARK-4495] Fix memory leak in JobProgressListener

Repository: spark Updated Branches: refs/heads/master c3002c4a6 -> 04d462f64 [SPARK-4495] Fix memory leak in JobProgressListener This commit fixes a memory leak in JobProgressListener that I introduced in SPARK-2321 and adds a testing framework to ensure that it’s very difficult to inadver

spark git commit: [SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream

Repository: spark Updated Branches: refs/heads/master 73c8ea84a -> c3002c4a6 [SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream In class TransformedDStream: ```scala require(parents.length > 0, "List of DStreams to transform is em

spark git commit: [SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream

Repository: spark Updated Branches: refs/heads/branch-1.2 c4abb2eb4 -> a250ca369 [SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream In class TransformedDStream: ```scala require(parents.length > 0, "List of DStreams to transform i

spark git commit: [DOC][PySpark][Streaming] Fix docstring for sphinx

Repository: spark Updated Branches: refs/heads/branch-1.2 8ecabf4b7 -> c4abb2eb4 [DOC][PySpark][Streaming] Fix docstring for sphinx This commit should be merged for 1.2 release. cc tdas Author: Ken Takagiwa Closes #3311 from giwa/patch-3 and squashes the following commits: ab474a8 [Ken Tak

spark git commit: [SPARK-4384] [PySpark] improve sort spilling

Repository: spark Updated Branches: refs/heads/branch-1.2 633d67cb7 -> 8ecabf4b7 [SPARK-4384] [PySpark] improve sort spilling If there some big broadcasts (or other object) in Python worker, the free memory could be used for sorting will be too small, then it will keep spilling small files i

spark git commit: [SPARK-4384] [PySpark] improve sort spilling

Repository: spark Updated Branches: refs/heads/master f9adda9af -> 73c8ea84a [SPARK-4384] [PySpark] improve sort spilling If there some big broadcasts (or other object) in Python worker, the free memory could be used for sorting will be too small, then it will keep spilling small files into

spark git commit: [SPARK-4429][BUILD] Build for Scala 2.11 using sbt fails.

Repository: spark Updated Branches: refs/heads/branch-1.2 fc73171d5 -> 633d67cb7 [SPARK-4429][BUILD] Build for Scala 2.11 using sbt fails. I tried to build for Scala 2.11 using sbt with the following command: ``` $ sbt/sbt -Dscala-2.11 assembly ``` but it ends with the following error messag

spark git commit: [SPARK-4429][BUILD] Build for Scala 2.11 using sbt fails.

Repository: spark Updated Branches: refs/heads/master 9b7bbcef8 -> f9adda9af [SPARK-4429][BUILD] Build for Scala 2.11 using sbt fails. I tried to build for Scala 2.11 using sbt with the following command: ``` $ sbt/sbt -Dscala-2.11 assembly ``` but it ends with the following error messages:

spark git commit: [DOC][PySpark][Streaming] Fix docstring for sphinx

Repository: spark Updated Branches: refs/heads/master 1c938413b -> 9b7bbcef8 [DOC][PySpark][Streaming] Fix docstring for sphinx This commit should be merged for 1.2 release. cc tdas Author: Ken Takagiwa Closes #3311 from giwa/patch-3 and squashes the following commits: ab474a8 [Ken Takagiw

spark git commit: SPARK-3962 Marked scope as provided for external projects.

Repository: spark Updated Branches: refs/heads/branch-1.2 ce5ea0fd6 -> fc73171d5 SPARK-3962 Marked scope as provided for external projects. Somehow maven shade plugin is set in infinite loop of creating effective pom. Author: Prashant Sharma Author: Prashant Sharma Closes #2959 from ScrapC

spark git commit: SPARK-3962 Marked scope as provided for external projects.

Repository: spark Updated Branches: refs/heads/master 0df02ca46 -> 1c938413b SPARK-3962 Marked scope as provided for external projects. Somehow maven shade plugin is set in infinite loop of creating effective pom. Author: Prashant Sharma Author: Prashant Sharma Closes #2959 from ScrapCodes

spark git commit: [HOT FIX] MiMa tests are broken

Repository: spark Updated Branches: refs/heads/master 3bf7ceebb -> 0df02ca46 [HOT FIX] MiMa tests are broken This is blocking #3353 and other patches. Author: Andrew Or Closes #3371 from andrewor14/mima-hot-fix and squashes the following commits: 842d059 [Andrew Or] Move excludes to the ri

spark git commit: [SPARK-4481][Streaming][Doc] Fix the wrong description of updateFunc

Repository: spark Updated Branches: refs/heads/master 22fc4e751 -> 3bf7ceebb [SPARK-4481][Streaming][Doc] Fix the wrong description of updateFunc Removed `If `this` function returns None, then corresponding state key-value pair will be eliminated.` for the description of `updateFunc: (Iterato

spark git commit: [SPARK-4482][Streaming] Disable ReceivedBlockTracker's write ahead log by default

Repository: spark Updated Branches: refs/heads/branch-1.2 2fb40e1aa -> ce5ea0fd6 [SPARK-4482][Streaming] Disable ReceivedBlockTracker's write ahead log by default The write ahead log of ReceivedBlockTracker gets enabled as soon as checkpoint directory is set. This should not happen, as the W

spark git commit: [SPARK-4482][Streaming] Disable ReceivedBlockTracker's write ahead log by default

Repository: spark Updated Branches: refs/heads/master eacc78834 -> 22fc4e751 [SPARK-4482][Streaming] Disable ReceivedBlockTracker's write ahead log by default The write ahead log of ReceivedBlockTracker gets enabled as soon as checkpoint directory is set. This should not happen, as the WAL s

spark git commit: [SPARK-4470] Validate number of threads in local mode

Repository: spark Updated Branches: refs/heads/branch-1.2 9da71f865 -> 2fb40e1aa [SPARK-4470] Validate number of threads in local mode When running Spark locally, if number of threads is specified as 0 (e.g., `spark-submit --master local[0] ...`), the job got stuck and does not run at all. I

spark git commit: [maven-release-plugin] prepare for next development iteration

Repository: spark Updated Branches: refs/heads/branch-1.1 3693ae5d3 -> 1df1c1d9b [maven-release-plugin] prepare for next development iteration Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1df1c1d9 Tree: http://git-wip-u

spark git commit: [maven-release-plugin] prepare release v1.1.1-rc2

Repository: spark Updated Branches: refs/heads/branch-1.1 aa3c794f8 -> 3693ae5d3 [maven-release-plugin] prepare release v1.1.1-rc2 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3693ae5d Tree: http://git-wip-us.apache.org

Git Push Summary

Repository: spark Updated Tags: refs/tags/v1.1.1-rc2 [created] 5ec7cdd87 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-4470] Validate number of threads in local mode

Repository: spark Updated Branches: refs/heads/master d75579d09 -> eacc78834 [SPARK-4470] Validate number of threads in local mode When running Spark locally, if number of threads is specified as 0 (e.g., `spark-submit --master local[0] ...`), the job got stuck and does not run at all. I thi

spark git commit: Update CHANGES.txt for 1.1.1-rc2

Repository: spark Updated Branches: refs/heads/branch-1.1 16bf5f3d1 -> aa3c794f8 Update CHANGES.txt for 1.1.1-rc2 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/aa3c794f Tree: http://git-wip-us.apache.org/repos/asf/spark/

spark git commit: [SPARK-4480] Avoid many small spills in external data structures (1.1)

Repository: spark Updated Branches: refs/heads/branch-1.1 e22a75923 -> 16bf5f3d1 [SPARK-4480] Avoid many small spills in external data structures (1.1) This is the branch-1.1 version of #3353. This requires a separate PR because the code in master has been refactored a little to eliminate dup

spark git commit: [SPARK-4467] fix elements read count for ExtrenalSorter

Repository: spark Updated Branches: refs/heads/master 5f5ac2daf -> d75579d09 [SPARK-4467] fix elements read count for ExtrenalSorter the elementsRead variable should be reset to 0 after each spilling Author: Tianshuo Deng Closes #3302 from tsdeng/fix_external_sorter_record_count and squashe

spark git commit: [SPARK-4467] fix elements read count for ExtrenalSorter

Repository: spark Updated Branches: refs/heads/branch-1.2 1d0fa7fb0 -> 9da71f865 [SPARK-4467] fix elements read count for ExtrenalSorter the elementsRead variable should be reset to 0 after each spilling Author: Tianshuo Deng Closes #3302 from tsdeng/fix_external_sorter_record_count and squ

spark git commit: SPARK-4455 Exclude dependency on hbase-annotations module

Repository: spark Updated Branches: refs/heads/branch-1.2 e0a20994f -> 1d0fa7fb0 SPARK-4455 Exclude dependency on hbase-annotations module pwendell Please take a look Author: tedyu Closes #3286 from tedyu/master and squashes the following commits: e61e610 [tedyu] SPARK-4455 Exclude depende

spark git commit: SPARK-4455 Exclude dependency on hbase-annotations module

Repository: spark Updated Branches: refs/heads/master 8327df697 -> 5f5ac2daf SPARK-4455 Exclude dependency on hbase-annotations module pwendell Please take a look Author: tedyu Closes #3286 from tedyu/master and squashes the following commits: e61e610 [tedyu] SPARK-4455 Exclude dependency

spark git commit: MAINTENANCE: Automated closing of pull requests.

Repository: spark Updated Branches: refs/heads/master 165cec9c4 -> 8327df697 MAINTENANCE: Automated closing of pull requests. This commit exists to close the following pull requests on Github: Closes #2777 (close requested by 'ankurdave') Closes #2947 (close requested by 'nchammas') Closes #3