spark git commit: [SPARK-12252][SPARK-12131][SQL] refactor MapObjects to make it less hacky

2015-12-09 Thread lian
Repository: spark Updated Branches: refs/heads/master bd2cd4f53 -> d8ec081c9 [SPARK-12252][SPARK-12131][SQL] refactor MapObjects to make it less hacky in https://github.com/apache/spark/pull/10133 we found that, we shoud ensure the children of `TreeNode` are all accessible in the `productIter

[1/2] spark git commit: [SPARK-12244][SPARK-12245][STREAMING] Rename trackStateByKey to mapWithState and change tracking function signature

2015-12-09 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-1.6 699f497cf -> f6d866173 http://git-wip-us.apache.org/repos/asf/spark/blob/f6d86617/streaming/src/test/java/org/apache/spark/streaming/JavaTrackStateByKeySuite.java -- dif

[2/2] spark git commit: [SPARK-12244][SPARK-12245][STREAMING] Rename trackStateByKey to mapWithState and change tracking function signature

2015-12-09 Thread zsxwing
[SPARK-12244][SPARK-12245][STREAMING] Rename trackStateByKey to mapWithState and change tracking function signature SPARK-12244: Based on feedback from early users and personal experience attempting to explain it, the name trackStateByKey had two problem. "trackState" is a completely new term w

[2/2] spark git commit: [SPARK-12244][SPARK-12245][STREAMING] Rename trackStateByKey to mapWithState and change tracking function signature

2015-12-09 Thread zsxwing
[SPARK-12244][SPARK-12245][STREAMING] Rename trackStateByKey to mapWithState and change tracking function signature SPARK-12244: Based on feedback from early users and personal experience attempting to explain it, the name trackStateByKey had two problem. "trackState" is a completely new term w

[1/2] spark git commit: [SPARK-12244][SPARK-12245][STREAMING] Rename trackStateByKey to mapWithState and change tracking function signature

2015-12-09 Thread zsxwing
Repository: spark Updated Branches: refs/heads/master 2166c2a75 -> bd2cd4f53 http://git-wip-us.apache.org/repos/asf/spark/blob/bd2cd4f5/streaming/src/test/java/org/apache/spark/streaming/JavaTrackStateByKeySuite.java -- diff --

spark git commit: [SPARK-11796] Fix httpclient and httpcore depedency issues related to docker-client

2015-12-09 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master ac8cdf1cd -> 2166c2a75 [SPARK-11796] Fix httpclient and httpcore depedency issues related to docker-client This commit fixes dependency issues which prevented the Docker-based JDBC integration tests from running in the Maven build. Autho

spark git commit: [SPARK-11796] Fix httpclient and httpcore depedency issues related to docker-client

2015-12-09 Thread joshrosen
Repository: spark Updated Branches: refs/heads/branch-1.6 9fe8dc916 -> 699f497cf [SPARK-11796] Fix httpclient and httpcore depedency issues related to docker-client This commit fixes dependency issues which prevented the Docker-based JDBC integration tests from running in the Maven build. A

spark git commit: [SPARK-11678][SQL][DOCS] Document basePath in the programming guide.

2015-12-09 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-1.6 d86a88da6 -> 9fe8dc916 [SPARK-11678][SQL][DOCS] Document basePath in the programming guide. This PR adds document for `basePath`, which is a new parameter used by `HadoopFsRelation`. The compiled doc is shown below. ![image](https://c

spark git commit: [SPARK-11678][SQL][DOCS] Document basePath in the programming guide.

2015-12-09 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 8770bd121 -> ac8cdf1cd [SPARK-11678][SQL][DOCS] Document basePath in the programming guide. This PR adds document for `basePath`, which is a new parameter used by `HadoopFsRelation`. The compiled doc is shown below. ![image](https://cloud

spark git commit: [SPARK-12165][ADDENDUM] Fix outdated comments on unroll test

2015-12-09 Thread joshrosen
Repository: spark Updated Branches: refs/heads/master 7a8e587dc -> 8770bd121 [SPARK-12165][ADDENDUM] Fix outdated comments on unroll test JoshRosen Author: Andrew Or Closes #10229 from andrewor14/unroll-test-comments. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http

spark git commit: [SPARK-12165][ADDENDUM] Fix outdated comments on unroll test

2015-12-09 Thread joshrosen
Repository: spark Updated Branches: refs/heads/branch-1.6 9bc6a27fd -> d86a88da6 [SPARK-12165][ADDENDUM] Fix outdated comments on unroll test JoshRosen Author: Andrew Or Closes #10229 from andrewor14/unroll-test-comments. (cherry picked from commit 8770bd1213f9b1051dabde9c5424ae7b32143a44)

spark git commit: [SPARK-12211][DOC][GRAPHX] Fix version number in graphx doc for migration from 1.1

2015-12-09 Thread jkbradley
Repository: spark Updated Branches: refs/heads/master 051c6a066 -> 7a8e587dc [SPARK-12211][DOC][GRAPHX] Fix version number in graphx doc for migration from 1.1 Migration from 1.1 section added to the GraphX doc in 1.2.0 (see https://spark.apache.org/docs/1.2.0/graphx-programming-guide.html#m

spark git commit: [SPARK-12211][DOC][GRAPHX] Fix version number in graphx doc for migration from 1.1

2015-12-09 Thread jkbradley
Repository: spark Updated Branches: refs/heads/branch-1.6 bfb420139 -> 9bc6a27fd [SPARK-12211][DOC][GRAPHX] Fix version number in graphx doc for migration from 1.1 Migration from 1.1 section added to the GraphX doc in 1.2.0 (see https://spark.apache.org/docs/1.2.0/graphx-programming-guide.ht

[2/3] spark git commit: [SPARK-11551][DOC] Replace example code in ml-features.md using include_example

2015-12-09 Thread meng
http://git-wip-us.apache.org/repos/asf/spark/blob/bfb42013/examples/src/main/java/org/apache/spark/examples/ml/JavaPolynomialExpansionExample.java -- diff --git a/examples/src/main/java/org/apache/spark/examples/ml/JavaPolynomialE

[3/3] spark git commit: [SPARK-11551][DOC] Replace example code in ml-features.md using include_example

2015-12-09 Thread meng
[SPARK-11551][DOC] Replace example code in ml-features.md using include_example PR on behalf of somideshmukh, thanks! Author: Xusen Yin Author: somideshmukh Closes #10219 from yinxusen/SPARK-11551. (cherry picked from commit 051c6a066f7b5fcc7472412144c15b50a5319bd5) Signed-off-by: Xiangrui Me

[3/3] spark git commit: [SPARK-11551][DOC] Replace example code in ml-features.md using include_example

2015-12-09 Thread meng
[SPARK-11551][DOC] Replace example code in ml-features.md using include_example PR on behalf of somideshmukh, thanks! Author: Xusen Yin Author: somideshmukh Closes #10219 from yinxusen/SPARK-11551. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.or

[1/3] spark git commit: [SPARK-11551][DOC] Replace example code in ml-features.md using include_example

2015-12-09 Thread meng
Repository: spark Updated Branches: refs/heads/branch-1.6 ee0a6e722 -> bfb420139 http://git-wip-us.apache.org/repos/asf/spark/blob/bfb42013/examples/src/main/scala/org/apache/spark/examples/ml/StringIndexerExample.scala -- diff

[2/3] spark git commit: [SPARK-11551][DOC] Replace example code in ml-features.md using include_example

2015-12-09 Thread meng
http://git-wip-us.apache.org/repos/asf/spark/blob/051c6a06/examples/src/main/java/org/apache/spark/examples/ml/JavaPolynomialExpansionExample.java -- diff --git a/examples/src/main/java/org/apache/spark/examples/ml/JavaPolynomialE

[1/3] spark git commit: [SPARK-11551][DOC] Replace example code in ml-features.md using include_example

2015-12-09 Thread meng
Repository: spark Updated Branches: refs/heads/master 1eb7c22ce -> 051c6a066 http://git-wip-us.apache.org/repos/asf/spark/blob/051c6a06/examples/src/main/scala/org/apache/spark/examples/ml/StringIndexerExample.scala -- diff --g

spark git commit: [SPARK-11824][WEBUI] WebUI does not render descriptions with 'bad' HTML, throws console error

2015-12-09 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.6 05e441e12 -> ee0a6e722 [SPARK-11824][WEBUI] WebUI does not render descriptions with 'bad' HTML, throws console error Don't warn when description isn't valid HTML since it may properly be like "SELECT ... where foo <= 1" The tests for

spark git commit: [SPARK-11824][WEBUI] WebUI does not render descriptions with 'bad' HTML, throws console error

2015-12-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master aec5ea000 -> 1eb7c22ce [SPARK-11824][WEBUI] WebUI does not render descriptions with 'bad' HTML, throws console error Don't warn when description isn't valid HTML since it may properly be like "SELECT ... where foo <= 1" The tests for thi

spark git commit: [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution

2015-12-09 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 442a7715a -> aec5ea000 [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution This patch fixes a bug in the eviction of storage memory by execution. ## The bug: In general, execution should be able to evict storage

spark git commit: [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution

2015-12-09 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.6 acd462420 -> 05e441e12 [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution This patch fixes a bug in the eviction of storage memory by execution. ## The bug: In general, execution should be able to evict sto

spark git commit: [SPARK-12241][YARN] Improve failure reporting in Yarn client obtainTokenForHBase()

2015-12-09 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 6900f0173 -> 442a7715a [SPARK-12241][YARN] Improve failure reporting in Yarn client obtainTokenForHBase() This lines up the HBase token logic with that done for Hive in SPARK-11265: reflection with only CFNE being swallowed. There is a t

spark git commit: [SPARK-10582][YARN][CORE] Fix AM failure situation for dynamic allocation

2015-12-09 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 22b9a8740 -> 6900f0173 [SPARK-10582][YARN][CORE] Fix AM failure situation for dynamic allocation Because of AM failure, the target executor number between driver and AM will be different, which will lead to unexpected behavior in dynamic a

spark git commit: [SPARK-10299][ML] word2vec should allow users to specify the window size

2015-12-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master 6e1c55eac -> 22b9a8740 [SPARK-10299][ML] word2vec should allow users to specify the window size Currently word2vec has the window hard coded at 5, some users may want different sizes (for example if using on n-gram input or similar). User

spark git commit: [SPARK-10299][ML] word2vec should allow users to specify the window size

2015-12-09 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.6 b5a76b4a4 -> acd462420 [SPARK-10299][ML] word2vec should allow users to specify the window size Currently word2vec has the window hard coded at 5, some users may want different sizes (for example if using on n-gram input or similar). U

spark git commit: [SPARK-12012][SQL] Show more comprehensive PhysicalRDD metadata when visualizing SQL query plan

2015-12-09 Thread rxin
Repository: spark Updated Branches: refs/heads/master a11321686 -> 6e1c55eac [SPARK-12012][SQL] Show more comprehensive PhysicalRDD metadata when visualizing SQL query plan This PR adds a `private[sql]` method `metadata` to `SparkPlan`, which can be used to describe detail information about

spark git commit: [SPARK-12031][CORE][BUG] Integer overflow when do sampling

2015-12-09 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-1.6 0be792aad -> b5a76b4a4 [SPARK-12031][CORE][BUG] Integer overflow when do sampling Author: uncleGen Closes #10023 from uncleGen/1.6-bugfix. (cherry picked from commit a113216865fd45ea39ae8f104e784af2cf667dcf) Signed-off-by: Sean Owen

spark git commit: [SPARK-12031][CORE][BUG] Integer overflow when do sampling

2015-12-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master f6883bb7a -> a11321686 [SPARK-12031][CORE][BUG] Integer overflow when do sampling Author: uncleGen Closes #10023 from uncleGen/1.6-bugfix. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/r