spark git commit: [SPARK-20284][CORE] Make {Des, S}erializationStream extend Closeable

2017-04-13 Thread srowen
Repository: spark Updated Branches: refs/heads/master 095d1cb3a -> a4293c284 [SPARK-20284][CORE] Make {Des,S}erializationStream extend Closeable ## What changes were proposed in this pull request? This PR allows to use `SerializationStream` and `DeserializationStream` in try-with-resources.

[2/2] spark-website git commit: regenerate html pointing to 2.11

2017-04-14 Thread srowen
regenerate html pointing to 2.11 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/fe9e3a88 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/fe9e3a88 Diff: http://git-wip-us.apache.org/repos/asf/sp

[1/2] spark-website git commit: add intellij information

2017-04-14 Thread srowen
Repository: spark-website Updated Branches: refs/heads/add_more_intellij_instructions [created] fe9e3a88c add intellij information Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/cbe2a9b8 Tree: http://git-w

[2/2] spark-website git commit: regenerate html pointing to 2.11

2017-04-14 Thread srowen
regenerate html pointing to 2.11 Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/fe9e3a88 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/fe9e3a88 Diff: http://git-wip-us.apache.org/repos/asf/sp

[1/2] spark-website git commit: add intellij information

2017-04-14 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site d39c4ecac -> fe9e3a88c add intellij information Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/cbe2a9b8 Tree: http://git-wip-us.apache.org/re

[spark-website] Git Push Summary

2017-04-14 Thread srowen
Repository: spark-website Updated Branches: refs/heads/add_more_intellij_instructions [deleted] fe9e3a88c - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.

spark git commit: [SPARK-20316][SQL] Val and Var should strictly follow the Scala syntax

2017-04-15 Thread srowen
Repository: spark Updated Branches: refs/heads/master fb036c441 -> 98b41ecbc [SPARK-20316][SQL] Val and Var should strictly follow the Scala syntax ## What changes were proposed in this pull request? val and var should strictly follow the Scala syntax ## How was this patch tested? manual te

spark git commit: [SPARK-19740][MESOS] Add support in Spark to pass arbitrary parameters into docker when running on mesos with docker containerizer

2017-04-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master e090f3c0c -> a888fed30 [SPARK-19740][MESOS] Add support in Spark to pass arbitrary parameters into docker when running on mesos with docker containerizer ## What changes were proposed in this pull request? Allow passing in arbitrary param

spark git commit: [SPARK-20343][BUILD] Add avro dependency in core POM to resolve build failure in SBT Hadoop 2.6 master on Jenkins

2017-04-16 Thread srowen
Repository: spark Updated Branches: refs/heads/master a888fed30 -> ad935f526 [SPARK-20343][BUILD] Add avro dependency in core POM to resolve build failure in SBT Hadoop 2.6 master on Jenkins ## What changes were proposed in this pull request? This PR proposes to add ``` org.a

[1/3] spark-website git commit: Alternate commit of PR 44 -- add new Spark Summits

2017-04-18 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site fe9e3a88c -> cc799c06f http://git-wip-us.apache.org/repos/asf/spark-website/blob/cc799c06/site/releases/spark-release-1-6-3.html -- diff --git a/site/releases/spar

[3/3] spark-website git commit: Alternate commit of PR 44 -- add new Spark Summits

2017-04-18 Thread srowen
Alternate commit of PR 44 -- add new Spark Summits Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/cc799c06 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/cc799c06 Diff: http://git-wip-us.apach

[2/3] spark-website git commit: Alternate commit of PR 44 -- add new Spark Summits

2017-04-18 Thread srowen
http://git-wip-us.apache.org/repos/asf/spark-website/blob/cc799c06/site/news/spark-and-shark-in-the-news.html -- diff --git a/site/news/spark-and-shark-in-the-news.html b/site/news/spark-and-shark-in-the-news.html index 714fe85..9

spark-website git commit: Oops, forgot to add new generated HTML post

2017-04-18 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site cc799c06f -> aa1c66e42 Oops, forgot to add new generated HTML post Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/aa1c66e4 Tree: http://git-w

spark git commit: [SPARK-20344][SCHEDULER] Duplicate call in FairSchedulableBuilder.addTaskSetManager

2017-04-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master b0a1e93e9 -> 07fd94e0d [SPARK-20344][SCHEDULER] Duplicate call in FairSchedulableBuilder.addTaskSetManager ## What changes were proposed in this pull request? Eliminate the duplicate call to `Pool.getSchedulableByName()` in `FairSchedula

spark git commit: [SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop 2.6 master on Jenkins

2017-04-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master 07fd94e0d -> d4f10cbbe [SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop 2.6 master on Jenkins ## What changes were proposed in this pull request? This PR proposes to force Avro's version to 1.7.7 i

spark git commit: [SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build

2017-04-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 773754b6c -> 35378766a [SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build ## What changes were proposed in this pull request? This PR proposes two things as below: - Avoid Unidoc build only if Hadoo

spark git commit: [SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build

2017-04-19 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 a87e21dd2 -> 8baa970bc [SPARK-20343][BUILD] Avoid Unidoc build only if Hadoop 2.6 is explicitly set in SBT build ## What changes were proposed in this pull request? This PR proposes two things as below: - Avoid Unidoc build only if H

spark git commit: [SPARK-20036][DOC] Note incompatible dependencies on org.apache.kafka artifacts

2017-04-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 35378766a -> 71a8e9df1 [SPARK-20036][DOC] Note incompatible dependencies on org.apache.kafka artifacts ## What changes were proposed in this pull request? Note that you shouldn't manually add dependencies on org.apache.kafka artifacts ##

spark git commit: [SPARK-20036][DOC] Note incompatible dependencies on org.apache.kafka artifacts

2017-04-19 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 8baa970bc -> 80a60da8f [SPARK-20036][DOC] Note incompatible dependencies on org.apache.kafka artifacts ## What changes were proposed in this pull request? Note that you shouldn't manually add dependencies on org.apache.kafka artifacts

spark git commit: [SPARK-20156][SQL][FOLLOW-UP] Java String toLowerCase "Turkish locale bug" in Database and Table DDLs

2017-04-20 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 8d658b90b -> d01122dbc [SPARK-20156][SQL][FOLLOW-UP] Java String toLowerCase "Turkish locale bug" in Database and Table DDLs ### What changes were proposed in this pull request? Database and Table names conform the Hive standard ("[a-z

spark git commit: [SPARK-20156][SQL][FOLLOW-UP] Java String toLowerCase "Turkish locale bug" in Database and Table DDLs

2017-04-20 Thread srowen
Repository: spark Updated Branches: refs/heads/master 46c574976 -> 55bea5691 [SPARK-20156][SQL][FOLLOW-UP] Java String toLowerCase "Turkish locale bug" in Database and Table DDLs ### What changes were proposed in this pull request? Database and Table names conform the Hive standard ("[a-zA-z_

spark git commit: Small rewording about history server use case

2017-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master e2b3d2367 -> 34767997e Small rewording about history server use case Hello PR #10991 removed the built-in history view from Spark Standalone, so the history server is no longer useful to Yarn or Mesos only. Author: Hervé Closes #17709

spark git commit: Small rewording about history server use case

2017-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 66e7a8f1d -> fb0351a3f Small rewording about history server use case Hello PR #10991 removed the built-in history view from Spark Standalone, so the history server is no longer useful to Yarn or Mesos only. Author: Hervé Closes #17

spark git commit: Small rewording about history server use case

2017-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 cddb4b7db -> eb4d097c3 Small rewording about history server use case Hello PR #10991 removed the built-in history view from Spark Standalone, so the history server is no longer useful to Yarn or Mesos only. Author: Hervé Closes #17

spark git commit: [SPARK-20401][DOC] In the spark official configuration document, the 'spark.driver.supervise' configuration parameter specification and default values are necessary.

2017-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/master fd648bff6 -> ad290402a [SPARK-20401][DOC] In the spark official configuration document, the 'spark.driver.supervise' configuration parameter specification and default values are necessary. ## What changes were proposed in this pull reques

spark git commit: [SPARK-20401][DOC] In the spark official configuration document, the 'spark.driver.supervise' configuration parameter specification and default values are necessary.

2017-04-21 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 ff1f989f2 -> 6c2489c66 [SPARK-20401][DOC] In the spark official configuration document, the 'spark.driver.supervise' configuration parameter specification and default values are necessary. ## What changes were proposed in this pull re

spark git commit: [SPARK-20386][SPARK CORE] modify the log info if the block exists on the slave already

2017-04-22 Thread srowen
Repository: spark Updated Branches: refs/heads/master ad290402a -> 05a451491 [SPARK-20386][SPARK CORE] modify the log info if the block exists on the slave already ## What changes were proposed in this pull request? Modify the added memory size to memSize-originalMemSize if the block exists

spark git commit: [SPARK-20386][SPARK CORE] modify the log info if the block exists on the slave already

2017-04-22 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 6c2489c66 -> d68e0a3a5 [SPARK-20386][SPARK CORE] modify the log info if the block exists on the slave already ## What changes were proposed in this pull request? Modify the added memory size to memSize-originalMemSize if the block exi

spark git commit: [SPARK-20385][WEB-UI] Submitted Time' field, the date format needs to be formatted, in running Drivers table or Completed Drivers table in master web ui.

2017-04-23 Thread srowen
Repository: spark Updated Branches: refs/heads/master 8765bc17d -> 2eaf4f3fe [SPARK-20385][WEB-UI] Submitted Time' field, the date format needs to be formatted, in running Drivers table or Completed Drivers table in master web ui. ## What changes were proposed in this pull request? Submitted

spark git commit: [SPARK-20385][WEB-UI] Submitted Time' field, the date format needs to be formatted, in running Drivers table or Completed Drivers table in master web ui.

2017-04-23 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 807c71892 -> cad33a730 [SPARK-20385][WEB-UI] Submitted Time' field, the date format needs to be formatted, in running Drivers table or Completed Drivers table in master web ui. ## What changes were proposed in this pull request? Submit

spark git commit: [BUILD] Close stale PRs

2017-04-24 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2eaf4f3fe -> e9f97154b [BUILD] Close stale PRs ## What changes were proposed in this pull request? This pr proposed to close stale PRs. Currently, we have 400+ open PRs and there are some stale PRs whose JIRA tickets have been already clos

[2/5] spark-website git commit: adjust the content structure to make it more reasonable

2017-04-25 Thread srowen
adjust the content structure to make it more reasonable Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/4e458563 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/4e458563 Diff: http://git-wip-us.

[4/5] spark-website git commit: Merge branch 'asf-site' of https://github.com/apache/spark-website into add-sbt-package

2017-04-25 Thread srowen
Merge branch 'asf-site' of https://github.com/apache/spark-website into add-sbt-package Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/3c96a509 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/

[1/5] spark-website git commit: fix an error in the descriptions of `Build Targets For Individual Projects`

2017-04-25 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site aa1c66e42 -> 09046892b fix an error in the descriptions of `Build Targets For Individual Projects` Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/co

[3/5] spark-website git commit: simplify maven build

2017-04-25 Thread srowen
simplify maven build Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/9902531e Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/9902531e Diff: http://git-wip-us.apache.org/repos/asf/spark-website/

[5/5] spark-website git commit: fix conflict

2017-04-25 Thread srowen
fix conflict Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/09046892 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/09046892 Diff: http://git-wip-us.apache.org/repos/asf/spark-website/diff/090

spark git commit: [SPARK-20455][DOCS] Fix Broken Docker IT Docs

2017-04-25 Thread srowen
Repository: spark Updated Branches: refs/heads/master 31345fde8 -> c8f121951 [SPARK-20455][DOCS] Fix Broken Docker IT Docs ## What changes were proposed in this pull request? Just added the Maven `test`goal. ## How was this patch tested? No test needed, just a trivial documentation fix. Au

spark git commit: [SPARK-20455][DOCS] Fix Broken Docker IT Docs

2017-04-25 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 fb59a1954 -> c18de9c04 [SPARK-20455][DOCS] Fix Broken Docker IT Docs ## What changes were proposed in this pull request? Just added the Maven `test`goal. ## How was this patch tested? No test needed, just a trivial documentation fix.

spark git commit: [SPARK-20455][DOCS] Fix Broken Docker IT Docs

2017-04-25 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 427966597 -> 65990fc57 [SPARK-20455][DOCS] Fix Broken Docker IT Docs ## What changes were proposed in this pull request? Just added the Maven `test`goal. ## How was this patch tested? No test needed, just a trivial documentation fix.

spark git commit: [SPARK-20404][CORE] Using Option(name) instead of Some(name)

2017-04-25 Thread srowen
Repository: spark Updated Branches: refs/heads/master c8f121951 -> 0bc7a9021 [SPARK-20404][CORE] Using Option(name) instead of Some(name) Using Option(name) instead of Some(name) to prevent runtime failures when using accumulators created like the following ``` sparkContext.accumulator(0, nul

spark git commit: [SPARK-20404][CORE] Using Option(name) instead of Some(name)

2017-04-25 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 65990fc57 -> 2d47e1aaf [SPARK-20404][CORE] Using Option(name) instead of Some(name) Using Option(name) instead of Some(name) to prevent runtime failures when using accumulators created like the following ``` sparkContext.accumulator(0,

spark git commit: [SPARK-20404][CORE] Using Option(name) instead of Some(name)

2017-04-25 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 c18de9c04 -> b62ebd91b [SPARK-20404][CORE] Using Option(name) instead of Some(name) Using Option(name) instead of Some(name) to prevent runtime failures when using accumulators created like the following ``` sparkContext.accumulator(0,

spark git commit: [SPARK-20400][DOCS] Remove References to 3rd Party Vendor Tools

2017-04-26 Thread srowen
Repository: spark Updated Branches: refs/heads/master df58a95a3 -> 7a365257e [SPARK-20400][DOCS] Remove References to 3rd Party Vendor Tools ## What changes were proposed in this pull request? Simple documentation change to remove explicit vendor references. ## How was this patch tested? NA

spark git commit: [SPARK-20400][DOCS] Remove References to 3rd Party Vendor Tools

2017-04-26 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 c8803c068 -> a2f5ced32 [SPARK-20400][DOCS] Remove References to 3rd Party Vendor Tools ## What changes were proposed in this pull request? Simple documentation change to remove explicit vendor references. ## How was this patch tested?

spark git commit: [SPARK-20465][CORE] Throws a proper exception when any temp directory could not be got

2017-04-28 Thread srowen
Repository: spark Updated Branches: refs/heads/master 59e3a5644 -> 8c911adac [SPARK-20465][CORE] Throws a proper exception when any temp directory could not be got ## What changes were proposed in this pull request? This PR proposes to throw an exception with better message rather than `Arr

spark git commit: [SPARK-20465][CORE] Throws a proper exception when any temp directory could not be got

2017-04-28 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 af3a1411a -> ea5b11446 [SPARK-20465][CORE] Throws a proper exception when any temp directory could not be got ## What changes were proposed in this pull request? This PR proposes to throw an exception with better message rather than

spark git commit: [SPARK-20521][DOC][CORE] The default of 'spark.worker.cleanup.appDataTtl' should be 604800 in spark-standalone.md

2017-04-30 Thread srowen
Repository: spark Updated Branches: refs/heads/master d228cd0b0 -> 4d99b95ad [SPARK-20521][DOC][CORE] The default of 'spark.worker.cleanup.appDataTtl' should be 604800 in spark-standalone.md ## What changes were proposed in this pull request? Currently, our project needs to be set to clean u

spark git commit: [SPARK-20521][DOC][CORE] The default of 'spark.worker.cleanup.appDataTtl' should be 604800 in spark-standalone.md

2017-04-30 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 9789d5c57 -> c5f559315 [SPARK-20521][DOC][CORE] The default of 'spark.worker.cleanup.appDataTtl' should be 604800 in spark-standalone.md ## What changes were proposed in this pull request? Currently, our project needs to be set to cle

spark git commit: [SPARK-20523][BUILD] Clean up build warnings for 2.2.0 release

2017-05-03 Thread srowen
How was this patch tested? Existing tests Author: Sean Owen Closes #17803 from srowen/SPARK-20523. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/16fab6b0 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/16fab6b0 D

spark git commit: [SPARK-20523][BUILD] Clean up build warnings for 2.2.0 release

2017-05-03 Thread srowen
ems ## How was this patch tested? Existing tests Author: Sean Owen Closes #17803 from srowen/SPARK-20523. (cherry picked from commit 16fab6b0ef3dcb33f92df30e17680922ad5fb672) Signed-off-by: Sean Owen Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.

spark git commit: [SPARK-16957][MLLIB] Use midpoints for split values.

2017-05-03 Thread srowen
Repository: spark Updated Branches: refs/heads/master 16fab6b0e -> 7f96f2d7f [SPARK-16957][MLLIB] Use midpoints for split values. ## What changes were proposed in this pull request? Use midpoints for split values now, and maybe later to make it weighted. ## How was this patch tested? + [x]

spark git commit: [INFRA] Close stale PRs

2017-05-05 Thread srowen
Repository: spark Updated Branches: refs/heads/master 0d16faab9 -> 4411ac705 [INFRA] Close stale PRs ## What changes were proposed in this pull request? This PR proposes to close a stale PR, several PRs suggested to be closed by a committer and obviously inappropriate PRs. Closes #9 Clo

spark git commit: [SPARK-19660][SQL] Replace the deprecated property name fs.default.name to fs.defaultFS that newly introduced

2017-05-05 Thread srowen
Repository: spark Updated Branches: refs/heads/master 4411ac705 -> 37cdf077c [SPARK-19660][SQL] Replace the deprecated property name fs.default.name to fs.defaultFS that newly introduced ## What changes were proposed in this pull request? Replace the deprecated property name `fs.default.name

spark git commit: [SPARK-19660][SQL] Replace the deprecated property name fs.default.name to fs.defaultFS that newly introduced

2017-05-05 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 c8756288d -> 7cb566abc [SPARK-19660][SQL] Replace the deprecated property name fs.default.name to fs.defaultFS that newly introduced ## What changes were proposed in this pull request? Replace the deprecated property name `fs.default.

spark git commit: [SPARK-20546][DEPLOY] spark-class gets syntax error in posix mode

2017-05-05 Thread srowen
Repository: spark Updated Branches: refs/heads/master 37cdf077c -> 5773ab121 [SPARK-20546][DEPLOY] spark-class gets syntax error in posix mode ## What changes were proposed in this pull request? Updated spark-class to turn off posix mode so the process substitution doesn't cause a syntax err

spark git commit: [SPARK-20546][DEPLOY] spark-class gets syntax error in posix mode

2017-05-05 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 7cb566abc -> dbb54a7b3 [SPARK-20546][DEPLOY] spark-class gets syntax error in posix mode ## What changes were proposed in this pull request? Updated spark-class to turn off posix mode so the process substitution doesn't cause a syntax

spark git commit: [SPARK-20546][DEPLOY] spark-class gets syntax error in posix mode

2017-05-05 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 d10b0f654 -> 179f5370e [SPARK-20546][DEPLOY] spark-class gets syntax error in posix mode ## What changes were proposed in this pull request? Updated spark-class to turn off posix mode so the process substitution doesn't cause a syntax

spark git commit: [SPARK-20518][CORE] Supplement the new blockidsuite unit tests

2017-05-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master 63d90e7da -> 37f963ac1 [SPARK-20518][CORE] Supplement the new blockidsuite unit tests ## What changes were proposed in this pull request? This PR adds the new unit tests to support ShuffleDataBlockId , ShuffleIndexBlockId , TempShuffleBlo

spark git commit: [SPARK-20484][MLLIB] Add documentation to ALS code

2017-05-07 Thread srowen
gxr srowen This contribution is my original work. I have the license to work on this project under the Spark project’s open source license. Author: Daniel Li Closes #17793 from danielyli/spark-20484. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.

spark git commit: [SPARK-7481][BUILD] Add spark-hadoop-cloud module to pull in object store access.

2017-05-07 Thread srowen
Repository: spark Updated Branches: refs/heads/master 88e6d7507 -> 2cf83c478 [SPARK-7481][BUILD] Add spark-hadoop-cloud module to pull in object store access. ## What changes were proposed in this pull request? Add a new `spark-hadoop-cloud` module and maven profile to pull in object store

spark git commit: [SPARK-20519][SQL][CORE] Modify to prevent some possible runtime exceptions

2017-05-08 Thread srowen
Repository: spark Updated Branches: refs/heads/master 2fdaeb52b -> 0f820e2b6 [SPARK-20519][SQL][CORE] Modify to prevent some possible runtime exceptions Signed-off-by: liuxian ## What changes were proposed in this pull request? When the input parameter is null, may be a runtime exception oc

spark git commit: [SPARK-20615][ML][TEST] SparseVector.argmax throws IndexOutOfBoundsException

2017-05-09 Thread srowen
Repository: spark Updated Branches: refs/heads/master 10b00abad -> be53a7835 [SPARK-20615][ML][TEST] SparseVector.argmax throws IndexOutOfBoundsException ## What changes were proposed in this pull request? Added a check for for the number of defined values. Previously the argmax function as

spark git commit: [SPARK-20615][ML][TEST] SparseVector.argmax throws IndexOutOfBoundsException

2017-05-09 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 a1112c615 -> f7a91a17e [SPARK-20615][ML][TEST] SparseVector.argmax throws IndexOutOfBoundsException ## What changes were proposed in this pull request? Added a check for for the number of defined values. Previously the argmax functio

spark git commit: [SPARK-20615][ML][TEST] SparseVector.argmax throws IndexOutOfBoundsException

2017-05-09 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 ca3f7edba -> 4bbfad44e [SPARK-20615][ML][TEST] SparseVector.argmax throws IndexOutOfBoundsException ## What changes were proposed in this pull request? Added a check for for the number of defined values. Previously the argmax functio

spark-website git commit: Direct 2.1.0, 2.0.1 downloads to archive; use https links for download; Apache Hadoop; remove stale download logic

2017-05-09 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site 7b32b181f -> b54c4f3fa Direct 2.1.0, 2.0.1 downloads to archive; use https links for download; Apache Hadoop; remove stale download logic Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-

spark-website git commit: Trigger git sync

2017-05-09 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site b54c4f3fa -> 01e0279a0 Trigger git sync Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/01e0279a Tree: http://git-wip-us.apache.org/repos/asf/

spark git commit: [SPARK-20630][WEB UI] Fixed column visibility in Executor Tab

2017-05-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master 804949c6b -> ca4625e0e [SPARK-20630][WEB UI] Fixed column visibility in Executor Tab ## What changes were proposed in this pull request? #14617 added new columns to the executor table causing the visibility checks for the logs and threadD

spark git commit: [SPARK-20630][WEB UI] Fixed column visibility in Executor Tab

2017-05-10 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 3ed2f4d51 -> 7597a522b [SPARK-20630][WEB UI] Fixed column visibility in Executor Tab ## What changes were proposed in this pull request? #14617 added new columns to the executor table causing the visibility checks for the logs and thr

spark git commit: [SPARK-20637][CORE] Remove mention of old RDD classes from comments

2017-05-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master ca4625e0e -> a4cbf26bc [SPARK-20637][CORE] Remove mention of old RDD classes from comments ## What changes were proposed in this pull request? A few comments around the code mention RDD classes that do not exist anymore. I'm not sure of t

spark git commit: [SPARK-20393][WEBU UI] Strengthen Spark to prevent XSS vulnerabilities

2017-05-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master a4cbf26bc -> b512233a4 [SPARK-20393][WEBU UI] Strengthen Spark to prevent XSS vulnerabilities ## What changes were proposed in this pull request? Add stripXSS and stripXSSMap to Spark Core's UIUtils. Calling these functions at any point t

spark git commit: [MINOR][BUILD] Fix lint-java breaks.

2017-05-10 Thread srowen
Repository: spark Updated Branches: refs/heads/master 76e4a5566 -> fcb88f921 [MINOR][BUILD] Fix lint-java breaks. ## What changes were proposed in this pull request? This PR proposes to fix the lint-breaks as below: ``` [ERROR] src/main/java/org/apache/spark/unsafe/Platform.java:[51] (regexp)

spark git commit: [MINOR][BUILD] Fix lint-java breaks.

2017-05-10 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 5f6029c75 -> 358516dcb [MINOR][BUILD] Fix lint-java breaks. ## What changes were proposed in this pull request? This PR proposes to fix the lint-breaks as below: ``` [ERROR] src/main/java/org/apache/spark/unsafe/Platform.java:[51] (reg

spark-website git commit: Trigger git sync

2017-05-10 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site 01e0279a0 -> c2c0905b4 Trigger git sync Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/c2c0905b Tree: http://git-wip-us.apache.org/repos/asf/

[2/2] spark-website git commit: Replace most http links with https as a best practice, where possible

2017-05-11 Thread srowen
Replace most http links with https as a best practice, where possible Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/62cf4a16 Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/62cf4a16 Diff: http

[1/2] spark-website git commit: Replace most http links with https as a best practice, where possible

2017-05-11 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site c2c0905b4 -> 62cf4a16d http://git-wip-us.apache.org/repos/asf/spark-website/blob/62cf4a16/site/sitemap.xml -- diff --git a/site/sitemap.xml b/site/sitemap.xml inde

spark-website git commit: More dead link fixing

2017-05-11 Thread srowen
Repository: spark-website Updated Branches: refs/heads/asf-site 62cf4a16d -> 5ed41c8d8 More dead link fixing Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/5ed41c8d Tree: http://git-wip-us.apache.org/repos

svn commit: r19618 - in /release/spark: spark-1.6.2/ spark-2.0.1/ spark-2.1.0/

2017-05-11 Thread srowen
Author: srowen Date: Thu May 11 10:08:00 2017 New Revision: 19618 Log: Delete non-current Spark releases Removed: release/spark/spark-1.6.2/ release/spark/spark-2.0.1/ release/spark/spark-2.1.0/ - To unsubscribe, e

spark git commit: [SPARK-20554][BUILD] Remove usage of scala.language.reflectiveCalls

2017-05-12 Thread srowen
bly resulting in more complex code. This turned out to be less significant than I thought, but, still worth a touch-up. ## How was this patch tested? Existing tests. Author: Sean Owen Closes #17949 from srowen/SPARK-20554. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-

spark git commit: [SPARK-20554][BUILD] Remove usage of scala.language.reflectiveCalls

2017-05-12 Thread srowen
bly resulting in more complex code. This turned out to be less significant than I thought, but, still worth a touch-up. ## How was this patch tested? Existing tests. Author: Sean Owen Closes #17949 from srowen/SPARK-20554. (cherry picked from commit fc8a2b6ee6f1041345f9ce9701fac496c3c3b1e6) Sig

spark git commit: [SPARK-20705][WEB-UI] The sort function can not be used in the master page when you use Firefox or Google Chrome.

2017-05-14 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 5842eeca5 -> cb64064dc [SPARK-20705][WEB-UI] The sort function can not be used in the master page when you use Firefox or Google Chrome. ## What changes were proposed in this pull request? When you open the master page, when you use Fi

spark git commit: [SPARK-20705][WEB-UI] The sort function can not be used in the master page when you use Firefox or Google Chrome.

2017-05-14 Thread srowen
Repository: spark Updated Branches: refs/heads/master 5a799fd8c -> 99d579992 [SPARK-20705][WEB-UI] The sort function can not be used in the master page when you use Firefox or Google Chrome. ## What changes were proposed in this pull request? When you open the master page, when you use Firefo

spark git commit: [SPARK-20705][WEB-UI] The sort function can not be used in the master page when you use Firefox or Google Chrome.

2017-05-14 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 95de4672e -> 62969e9be [SPARK-20705][WEB-UI] The sort function can not be used in the master page when you use Firefox or Google Chrome. ## What changes were proposed in this pull request? When you open the master page, when you use Fi

spark git commit: [SPARK-20720][WEB-UI] Executor Summary' should show the exact number, 'Removed Executors' should display the specific number, in the Application Page

2017-05-14 Thread srowen
Repository: spark Updated Branches: refs/heads/master 99d579992 -> 8da6e8b1f [SPARK-20720][WEB-UI] Executor Summary' should show the exact number, 'Removed Executors' should display the specific number, in the Application Page ## What changes were proposed in this pull request? When the numb

spark git commit: [SPARK-20769][DOC] Incorrect documentation for using Jupyter notebook

2017-05-17 Thread srowen
Repository: spark Updated Branches: refs/heads/master 30e0557db -> 199541769 [SPARK-20769][DOC] Incorrect documentation for using Jupyter notebook ## What changes were proposed in this pull request? SPARK-13973 incorrectly removed the required PYSPARK_DRIVER_PYTHON_OPTS=notebook from documen

spark git commit: [SPARK-20769][DOC] Incorrect documentation for using Jupyter notebook

2017-05-17 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 d42c67a1f -> dac0b50b6 [SPARK-20769][DOC] Incorrect documentation for using Jupyter notebook ## What changes were proposed in this pull request? SPARK-13973 incorrectly removed the required PYSPARK_DRIVER_PYTHON_OPTS=notebook from doc

spark git commit: [SPARK-20769][DOC] Incorrect documentation for using Jupyter notebook

2017-05-17 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 14b6a9d34 -> ba35c6bd3 [SPARK-20769][DOC] Incorrect documentation for using Jupyter notebook ## What changes were proposed in this pull request? SPARK-13973 incorrectly removed the required PYSPARK_DRIVER_PYTHON_OPTS=notebook from doc

spark git commit: [INFRA] Close stale PRs

2017-05-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master b7aac15d5 -> 5d2750aa2 [INFRA] Close stale PRs ## What changes were proposed in this pull request? This PR proposes to close PRs ... - inactive to the review comments more than a month - WIP and inactive more than a month - with Jen

spark git commit: [SPARK-20779][EXAMPLES] The ASF header placed in an incorrect location in some files.

2017-05-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master 5d2750aa2 -> 4779b86b5 [SPARK-20779][EXAMPLES] The ASF header placed in an incorrect location in some files. ## What changes were proposed in this pull request? The license is not at the top in some files. and it will be best if we update

spark git commit: [SPARK-20796] the location of start-master.sh in spark-standalone.md is wrong

2017-05-18 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 c708b1480 -> db821fe55 [SPARK-20796] the location of start-master.sh in spark-standalone.md is wrong [https://issues.apache.org/jira/browse/SPARK-20796](https://issues.apache.org/jira/browse/SPARK-20796) the location of start-master.sh

spark git commit: [SPARK-20796] the location of start-master.sh in spark-standalone.md is wrong

2017-05-18 Thread srowen
Repository: spark Updated Branches: refs/heads/master 4779b86b5 -> 99452df44 [SPARK-20796] the location of start-master.sh in spark-standalone.md is wrong [https://issues.apache.org/jira/browse/SPARK-20796](https://issues.apache.org/jira/browse/SPARK-20796) the location of start-master.sh in s

spark git commit: [SPARK-20796] the location of start-master.sh in spark-standalone.md is wrong

2017-05-18 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 ba35c6bd3 -> e06d9367f [SPARK-20796] the location of start-master.sh in spark-standalone.md is wrong [https://issues.apache.org/jira/browse/SPARK-20796](https://issues.apache.org/jira/browse/SPARK-20796) the location of start-master.sh

spark git commit: [SPARK-20607][CORE] Add new unit tests to ShuffleSuite

2017-05-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 3f2cd51ee -> f398640da [SPARK-20607][CORE] Add new unit tests to ShuffleSuite ## What changes were proposed in this pull request? This PR update to two: 1.adds the new unit tests. testing would be performed when there is no shuffle stage

spark git commit: [SPARK-20759] SCALA_VERSION in _config.yml should be consistent with pom.xml

2017-05-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master f398640da -> dba2ca2c1 [SPARK-20759] SCALA_VERSION in _config.yml should be consistent with pom.xml [https://issues.apache.org/jira/browse/SPARK-20759](https://issues.apache.org/jira/browse/SPARK-20759) SCALA_VERSION in _config.yml is 2.11.

spark git commit: [SPARK-20759] SCALA_VERSION in _config.yml should be consistent with pom.xml

2017-05-19 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 939b9536f -> 001b82c18 [SPARK-20759] SCALA_VERSION in _config.yml should be consistent with pom.xml [https://issues.apache.org/jira/browse/SPARK-20759](https://issues.apache.org/jira/browse/SPARK-20759) SCALA_VERSION in _config.yml is 2

spark git commit: [SPARK-20759] SCALA_VERSION in _config.yml should be consistent with pom.xml

2017-05-19 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 e326de485 -> c53fe793d [SPARK-20759] SCALA_VERSION in _config.yml should be consistent with pom.xml [https://issues.apache.org/jira/browse/SPARK-20759](https://issues.apache.org/jira/browse/SPARK-20759) SCALA_VERSION in _config.yml is 2

spark git commit: [SPARK-20781] the location of Dockerfile in docker.properties.templat is wrong

2017-05-19 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.1 c53fe793d -> e9804b3d4 [SPARK-20781] the location of Dockerfile in docker.properties.templat is wrong [https://issues.apache.org/jira/browse/SPARK-20781](https://issues.apache.org/jira/browse/SPARK-20781) the location of Dockerfile in d

spark git commit: [SPARK-20781] the location of Dockerfile in docker.properties.templat is wrong

2017-05-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master b5d8d9ba1 -> 749418d28 [SPARK-20781] the location of Dockerfile in docker.properties.templat is wrong [https://issues.apache.org/jira/browse/SPARK-20781](https://issues.apache.org/jira/browse/SPARK-20781) the location of Dockerfile in docke

spark git commit: [SPARK-20781] the location of Dockerfile in docker.properties.templat is wrong

2017-05-19 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 4fcd52b48 -> 3aad5982a [SPARK-20781] the location of Dockerfile in docker.properties.templat is wrong [https://issues.apache.org/jira/browse/SPARK-20781](https://issues.apache.org/jira/browse/SPARK-20781) the location of Dockerfile in d

spark git commit: [SPARK-20806][DEPLOY] Launcher: redundant check for Spark lib dir

2017-05-20 Thread srowen
ing tests Author: Sean Owen Closes #18032 from srowen/SPARK-20806. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bbd8d7de Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bbd8d7de Diff: http://git-wip-us.apache.

spark git commit: [SPARK-20687][MLLIB] mllib.Matrices.fromBreeze may crash when converting from Breeze sparse matrix

2017-05-22 Thread srowen
Repository: spark Updated Branches: refs/heads/master a2b3b6762 -> 06dda1d58 [SPARK-20687][MLLIB] mllib.Matrices.fromBreeze may crash when converting from Breeze sparse matrix ## What changes were proposed in this pull request? When two Breeze SparseMatrices are operated, the result matrix m

<    17   18   19   20   21   22   23   24   25   26   >