spark git commit: [SPARK-12103][STREAMING][KAFKA][DOC] document that K means Key and V …

Repository: spark Updated Branches: refs/heads/master 4a39b5a1b -> 48a9804b2 [SPARK-12103][STREAMING][KAFKA][DOC] document that K means Key and V … …means Value Author: cody koeninger Closes #10132 from koeninger/SPARK-12103. Project: http://git-wip-us.apache.org/repos/asf/spark/repo

spark git commit: [SPARK-12103][STREAMING][KAFKA][DOC] document that K means Key and V …

Repository: spark Updated Branches: refs/heads/branch-1.6 5c8216920 -> c9e5274ae [SPARK-12103][STREAMING][KAFKA][DOC] document that K means Key and V … …means Value Author: cody koeninger Closes #10132 from koeninger/SPARK-12103. (cherry picked from commit 48a9804b2ad89b3fb204c79f0dbad

spark git commit: [SPARK-12166][TEST] Unset hadoop related environment in testing

Repository: spark Updated Branches: refs/heads/master 48a9804b2 -> 708129187 [SPARK-12166][TEST] Unset hadoop related environment in testing Author: Jeff Zhang Closes #10172 from zjffdu/SPARK-12166. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache

spark git commit: [SPARK-12166][TEST] Unset hadoop related environment in testing

Repository: spark Updated Branches: refs/heads/branch-1.6 c9e5274ae -> 870f43562 [SPARK-12166][TEST] Unset hadoop related environment in testing Author: Jeff Zhang Closes #10172 from zjffdu/SPARK-12166. (cherry picked from commit 708129187a460aca30790281e9221c0cd5e271df) Signed-off-by: Sean

spark git commit: [SPARK-11439][ML] Optimization of creating sparse feature without dense one

Repository: spark Updated Branches: refs/heads/master 708129187 -> 037b7e76a [SPARK-11439][ML] Optimization of creating sparse feature without dense one Sparse feature generated in LinearDataGenerator does not create dense vectors as an intermediate any more. Author: Nakul Jindal Closes #9

[3/3] spark git commit: [SPARK-11551][DOC][EXAMPLE] Revert PR #10002

[SPARK-11551][DOC][EXAMPLE] Revert PR #10002 This reverts PR #10002, commit 78209b0ccaf3f22b5e2345dfb2b98edfdb746819. The original PR wasn't tested on Jenkins before being merged. Author: Cheng Lian Closes #10200 from liancheng/revert-pr-10002. Project: http://git-wip-us.apache.org/repos/asf

[2/3] spark git commit: [SPARK-11551][DOC][EXAMPLE] Revert PR #10002

http://git-wip-us.apache.org/repos/asf/spark/blob/da2012a0/examples/src/main/java/org/apache/spark/examples/ml/JavaRFormulaExample.java -- diff --git a/examples/src/main/java/org/apache/spark/examples/ml/JavaRFormulaExample.java

[1/3] spark git commit: [SPARK-11551][DOC][EXAMPLE] Revert PR #10002

Repository: spark Updated Branches: refs/heads/master 037b7e76a -> da2012a0e http://git-wip-us.apache.org/repos/asf/spark/blob/da2012a0/examples/src/main/scala/org/apache/spark/examples/ml/TokenizerExample.scala -- diff --git

[2/3] spark git commit: [SPARK-11551][DOC][EXAMPLE] Revert PR #10002

http://git-wip-us.apache.org/repos/asf/spark/blob/8a791a32/examples/src/main/java/org/apache/spark/examples/ml/JavaRFormulaExample.java -- diff --git a/examples/src/main/java/org/apache/spark/examples/ml/JavaRFormulaExample.java

[3/3] spark git commit: [SPARK-11551][DOC][EXAMPLE] Revert PR #10002

[SPARK-11551][DOC][EXAMPLE] Revert PR #10002 This reverts PR #10002, commit 78209b0ccaf3f22b5e2345dfb2b98edfdb746819. The original PR wasn't tested on Jenkins before being merged. Author: Cheng Lian Closes #10200 from liancheng/revert-pr-10002. (cherry picked from commit da2012a0e152aa078bdd1

[1/3] spark git commit: [SPARK-11551][DOC][EXAMPLE] Revert PR #10002

Repository: spark Updated Branches: refs/heads/branch-1.6 870f43562 -> 8a791a327 http://git-wip-us.apache.org/repos/asf/spark/blob/8a791a32/examples/src/main/scala/org/apache/spark/examples/ml/TokenizerExample.scala -- diff --g

spark git commit: [SPARK-11652][CORE] Remote code execution with InvokerTransformer

Repository: spark Updated Branches: refs/heads/master da2012a0e -> e3735ce16 [SPARK-11652][CORE] Remote code execution with InvokerTransformer Fix commons-collection group ID to commons-collections for version 3.x Patches earlier PR at https://github.com/apache/spark/pull/9731 Author: Sean O

spark git commit: [SPARK-11652][CORE] Remote code execution with InvokerTransformer

Repository: spark Updated Branches: refs/heads/branch-1.6 8a791a327 -> c8f9eb749 [SPARK-11652][CORE] Remote code execution with InvokerTransformer Fix commons-collection group ID to commons-collections for version 3.x Patches earlier PR at https://github.com/apache/spark/pull/9731 Author: Se

spark git commit: [SPARK-11652][CORE] Remote code execution with InvokerTransformer

Repository: spark Updated Branches: refs/heads/branch-1.4 b6ba2dab2 -> c7c99857d [SPARK-11652][CORE] Remote code execution with InvokerTransformer Fix commons-collection group ID to commons-collections for version 3.x Patches earlier PR at https://github.com/apache/spark/pull/9731 Author: Se

spark git commit: [SPARK-11652][CORE] Remote code execution with InvokerTransformer

Repository: spark Updated Branches: refs/heads/branch-1.5 2f30927a5 -> 4b99f72f7 [SPARK-11652][CORE] Remote code execution with InvokerTransformer Fix commons-collection group ID to commons-collections for version 3.x Patches earlier PR at https://github.com/apache/spark/pull/9731 Author: Se

spark git commit: [SPARK-11155][WEB UI] Stage summary json should include stage duration

Repository: spark Updated Branches: refs/heads/master e3735ce16 -> 6cb06e871 [SPARK-11155][WEB UI] Stage summary json should include stage duration The json endpoint for stages doesn't include information on the stage duration that is present in the UI. This looks like a simple oversight, the

spark git commit: [SPARK-12074] Avoid memory copy involving ByteBuffer.wrap(ByteArrayOutputStream.toByteArray)

Repository: spark Updated Branches: refs/heads/master 6cb06e871 -> 75c60bf4b [SPARK-12074] Avoid memory copy involving ByteBuffer.wrap(ByteArrayOutputStream.toByteArray) SPARK-12060 fixed JavaSerializerInstance.serialize This PR applies the same technique on two other classes. zsxwing Autho

spark git commit: [SPARK-12201][SQL] add type coercion rule for greatest/least

Repository: spark Updated Branches: refs/heads/master 75c60bf4b -> 381f17b54 [SPARK-12201][SQL] add type coercion rule for greatest/least checked with hive, greatest/least should cast their children to a tightest common type, i.e. `(int, long) => long`, `(int, string) => error`, `(decimal(10,

spark git commit: [SPARK-12201][SQL] add type coercion rule for greatest/least

Repository: spark Updated Branches: refs/heads/branch-1.6 c8f9eb749 -> 8ef33aa1f [SPARK-12201][SQL] add type coercion rule for greatest/least checked with hive, greatest/least should cast their children to a tightest common type, i.e. `(int, long) => long`, `(int, string) => error`, `(decimal

spark git commit: [SPARK-12195][SQL] Adding BigDecimal, Date and Timestamp into Encoder

Repository: spark Updated Branches: refs/heads/master 381f17b54 -> c0b13d556 [SPARK-12195][SQL] Adding BigDecimal, Date and Timestamp into Encoder This PR is to add three more data types into Encoder, including `BigDecimal`, `Date` and `Timestamp`. marmbrus cloud-fan rxin Could you take a qu

spark git commit: [SPARK-12195][SQL] Adding BigDecimal, Date and Timestamp into Encoder

Repository: spark Updated Branches: refs/heads/branch-1.6 8ef33aa1f -> 9eeb0f25a [SPARK-12195][SQL] Adding BigDecimal, Date and Timestamp into Encoder This PR is to add three more data types into Encoder, including `BigDecimal`, `Date` and `Timestamp`. marmbrus cloud-fan rxin Could you take

spark git commit: [SPARK-12188][SQL] Code refactoring and comment correction in Dataset APIs

Repository: spark Updated Branches: refs/heads/branch-1.6 9eeb0f25a -> be0fe9b45 [SPARK-12188][SQL] Code refactoring and comment correction in Dataset APIs This PR contains the following updates: - Created a new private variable `boundTEncoder` that can be shared by multiple functions, `RDD`

spark git commit: [SPARK-12188][SQL] Code refactoring and comment correction in Dataset APIs

Repository: spark Updated Branches: refs/heads/master c0b13d556 -> 5d96a710a [SPARK-12188][SQL] Code refactoring and comment correction in Dataset APIs This PR contains the following updates: - Created a new private variable `boundTEncoder` that can be shared by multiple functions, `RDD`, `s

spark git commit: [SPARK-10393] use ML pipeline in LDA example

Repository: spark Updated Branches: refs/heads/master 5d96a710a -> 872a2ee28 [SPARK-10393] use ML pipeline in LDA example jira: https://issues.apache.org/jira/browse/SPARK-10393 Since the logic of the text processing part has been moved to ML estimators/transformers, replace the related code

spark git commit: [SPARK-10393] use ML pipeline in LDA example

Repository: spark Updated Branches: refs/heads/branch-1.6 be0fe9b45 -> 1c8451b5e [SPARK-10393] use ML pipeline in LDA example jira: https://issues.apache.org/jira/browse/SPARK-10393 Since the logic of the text processing part has been moved to ML estimators/transformers, replace the related

spark git commit: [SPARK-12205][SQL] Pivot fails Analysis when aggregate is UnresolvedFunction

Repository: spark Updated Branches: refs/heads/master 872a2ee28 -> 4bcb89494 [SPARK-12205][SQL] Pivot fails Analysis when aggregate is UnresolvedFunction Delays application of ResolvePivot until all aggregates are resolved to prevent problems with UnresolvedFunction and adds unit test Author

spark git commit: [SPARK-12205][SQL] Pivot fails Analysis when aggregate is UnresolvedFunction

Repository: spark Updated Branches: refs/heads/branch-1.6 1c8451b5e -> 9145bfb81 [SPARK-12205][SQL] Pivot fails Analysis when aggregate is UnresolvedFunction Delays application of ResolvePivot until all aggregates are resolved to prevent problems with UnresolvedFunction and adds unit test Au

spark git commit: [SPARK-11605][MLLIB] ML 1.6 QA: API: Java compatibility, docs

Repository: spark Updated Branches: refs/heads/master 4bcb89494 -> 5cb469505 [SPARK-11605][MLLIB] ML 1.6 QA: API: Java compatibility, docs jira: https://issues.apache.org/jira/browse/SPARK-11605 Check Java compatibility for MLlib for this release. fix: 1. `StreamingTest.registerStream` needs

spark git commit: [SPARK-11605][MLLIB] ML 1.6 QA: API: Java compatibility, docs

Repository: spark Updated Branches: refs/heads/branch-1.6 9145bfb81 -> 7e45feb00 [SPARK-11605][MLLIB] ML 1.6 QA: API: Java compatibility, docs jira: https://issues.apache.org/jira/browse/SPARK-11605 Check Java compatibility for MLlib for this release. fix: 1. `StreamingTest.registerStream` n

spark git commit: [SPARK-12159][ML] Add user guide section for IndexToString transformer

Repository: spark Updated Branches: refs/heads/master 5cb469505 -> 06746b300 [SPARK-12159][ML] Add user guide section for IndexToString transformer Documentation regarding the `IndexToString` label transformer with code snippets in Scala/Java/Python. Author: BenFradet Closes #10166 from Be

spark git commit: [SPARK-12159][ML] Add user guide section for IndexToString transformer

Repository: spark Updated Branches: refs/heads/branch-1.6 7e45feb00 -> 3e31e7e24 [SPARK-12159][ML] Add user guide section for IndexToString transformer Documentation regarding the `IndexToString` label transformer with code snippets in Scala/Java/Python. Author: BenFradet Closes #10166 fro

spark git commit: [SPARK-3873][BUILD] Add style checker to enforce import ordering.

Repository: spark Updated Branches: refs/heads/master 06746b300 -> 2ff17bcfb [SPARK-3873][BUILD] Add style checker to enforce import ordering. The checker tries to follow as closely as possible the guidelines of the code style document, and makes some decisions where the guide is not clear. In

spark git commit: [SPARK-12187] *MemoryPool classes should not be fully public

Repository: spark Updated Branches: refs/heads/branch-1.6 3e31e7e24 -> 25249d1ec [SPARK-12187] *MemoryPool classes should not be fully public This patch tightens them to `private[memory]`. Author: Andrew Or Closes #10182 from andrewor14/memory-visibility. (cherry picked from commit 9494521

spark git commit: [SPARK-12187] *MemoryPool classes should not be fully public

Repository: spark Updated Branches: refs/heads/master 2ff17bcfb -> 949452169 [SPARK-12187] *MemoryPool classes should not be fully public This patch tightens them to `private[memory]`. Author: Andrew Or Closes #10182 from andrewor14/memory-visibility. Project: http://git-wip-us.apache.org

spark git commit: [SPARK-12069][SQL] Update documentation with Datasets

Repository: spark Updated Branches: refs/heads/master 949452169 -> 395948942 [SPARK-12069][SQL] Update documentation with Datasets Author: Michael Armbrust Closes #10060 from marmbrus/docs. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repo

spark git commit: [SPARK-12069][SQL] Update documentation with Datasets

Repository: spark Updated Branches: refs/heads/branch-1.6 25249d1ec -> 2a5e4d157 [SPARK-12069][SQL] Update documentation with Datasets Author: Michael Armbrust Closes #10060 from marmbrus/docs. (cherry picked from commit 39594894232e0b70c5ca8b0df137da0d61223fd5) Signed-off-by: Michael Armbr

spark git commit: [SPARK-8517][ML][DOC] Reorganizes the spark.ml user guide

Repository: spark Updated Branches: refs/heads/master 395948942 -> 765c67f5f [SPARK-8517][ML][DOC] Reorganizes the spark.ml user guide This PR moves pieces of the spark.ml user guide to reflect suggestions in SPARK-8517. It does not introduce new content, as requested. https://cloud.githubus

spark git commit: [SPARK-8517][ML][DOC] Reorganizes the spark.ml user guide

Repository: spark Updated Branches: refs/heads/branch-1.6 2a5e4d157 -> b1d5a7859 [SPARK-8517][ML][DOC] Reorganizes the spark.ml user guide This PR moves pieces of the spark.ml user guide to reflect suggestions in SPARK-8517. It does not introduce new content, as requested. https://cloud.gith

spark git commit: [SPARK-11343][ML] Documentation of float and double prediction/label columns in RegressionEvaluator

Repository: spark Updated Branches: refs/heads/master 765c67f5f -> a0046e379 [SPARK-11343][ML] Documentation of float and double prediction/label columns in RegressionEvaluator felixcheung , mengxr Just added a message to require() Author: Dominik Dahlem Closes #9598 from dahlem/ddahlem_

spark git commit: [SPARK-11343][ML] Documentation of float and double prediction/label columns in RegressionEvaluator

Repository: spark Updated Branches: refs/heads/branch-1.6 b1d5a7859 -> 9e82273af [SPARK-11343][ML] Documentation of float and double prediction/label columns in RegressionEvaluator felixcheung , mengxr Just added a message to require() Author: Dominik Dahlem Closes #9598 from dahlem/ddah

spark git commit: [SPARK-12222] [CORE] Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception

Repository: spark Updated Branches: refs/heads/master a0046e379 -> 3934562d3 [SPARK-1] [CORE] Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception Jira: https://issues.apache.org/jira/browse/SPARK-1 Deserialize RoaringBitmap using Kryo serializer throw Buf

spark git commit: [SPARK-12222] [CORE] Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception

Repository: spark Updated Branches: refs/heads/branch-1.6 9e82273af -> 0be792aad [SPARK-1] [CORE] Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception Jira: https://issues.apache.org/jira/browse/SPARK-1 Deserialize RoaringBitmap using Kryo serializer throw

spark git commit: [SPARK-11676][SQL] Parquet filter tests all pass if filters are not really pushed down

Repository: spark Updated Branches: refs/heads/master 3934562d3 -> f6883bb7a [SPARK-11676][SQL] Parquet filter tests all pass if filters are not really pushed down Currently Parquet predicate tests all pass even if filters are not pushed down or this is disabled. In this PR, For checking ev