spark git commit: [SPARKR][MINOR] Fix docs for sparkR.session and count

2016-09-02 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-2.0 c0ea77071 -> 12a2e2a5a [SPARKR][MINOR] Fix docs for sparkR.session and count ## What changes were proposed in this pull request? This PR tries to add some more explanation to `sparkR.session`. It also modifies doc for `count` so when

spark git commit: [SPARKR][MINOR] Fix docs for sparkR.session and count

2016-09-02 Thread shivaram
Repository: spark Updated Branches: refs/heads/master e6132a6cf -> d2fde6b72 [SPARKR][MINOR] Fix docs for sparkR.session and count ## What changes were proposed in this pull request? This PR tries to add some more explanation to `sparkR.session`. It also modifies doc for `count` so when

spark git commit: Revert "[SPARK-16334] Reusing same dictionary column for decoding consecutive row groups shouldn't throw an error"

2016-09-02 Thread davies
Repository: spark Updated Branches: refs/heads/branch-2.0 b8f65dad7 -> c0ea77071 Revert "[SPARK-16334] Reusing same dictionary column for decoding consecutive row groups shouldn't throw an error" This reverts commit a3930c3b9afa9f7eba2a5c8b8f279ca38e348e9b. Project:

spark git commit: Fix build

2016-09-02 Thread davies
Repository: spark Updated Branches: refs/heads/branch-2.0 a3930c3b9 -> b8f65dad7 Fix build Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b8f65dad Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b8f65dad Diff:

spark git commit: [SPARK-17298][SQL] Require explicit CROSS join for cartesian products

2016-09-02 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master a2c9acb0e -> e6132a6cf [SPARK-17298][SQL] Require explicit CROSS join for cartesian products ## What changes were proposed in this pull request? Require the use of CROSS join syntax in SQL (and a new crossJoin DataFrame API) to specify

spark git commit: [SPARK-16334] Reusing same dictionary column for decoding consecutive row groups shouldn't throw an error

2016-09-02 Thread davies
Repository: spark Updated Branches: refs/heads/master ed9c884dc -> a2c9acb0e [SPARK-16334] Reusing same dictionary column for decoding consecutive row groups shouldn't throw an error ## What changes were proposed in this pull request? This patch fixes a bug in the vectorized parquet reader

spark git commit: [SPARK-17230] [SQL] Should not pass optimized query into QueryExecution in DataFrameWriter

2016-09-02 Thread davies
Repository: spark Updated Branches: refs/heads/branch-2.0 03d9af604 -> c9c36fa0c [SPARK-17230] [SQL] Should not pass optimized query into QueryExecution in DataFrameWriter Some analyzer rules have assumptions on logical plans, optimizer may break these assumption, we should not pass an

spark git commit: [SPARK-17230] [SQL] Should not pass optimized query into QueryExecution in DataFrameWriter

2016-09-02 Thread davies
Repository: spark Updated Branches: refs/heads/master eac1d0e92 -> ed9c884dc [SPARK-17230] [SQL] Should not pass optimized query into QueryExecution in DataFrameWriter ## What changes were proposed in this pull request? Some analyzer rules have assumptions on logical plans, optimizer may

spark git commit: [SPARK-17316][CORE] Make CoarseGrainedSchedulerBackend.removeExecutor non-blocking

2016-09-02 Thread zsxwing
Repository: spark Updated Branches: refs/heads/branch-1.6 412b0e896 -> b84a92c24 [SPARK-17316][CORE] Make CoarseGrainedSchedulerBackend.removeExecutor non-blocking ## What changes were proposed in this pull request? StandaloneSchedulerBackend.executorRemoved is a blocking call right now. It

spark git commit: [SPARK-17376][SPARKR] followup - change since version

2016-09-02 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.0 d4ae35d02 -> 03d9af604 [SPARK-17376][SPARKR] followup - change since version ## What changes were proposed in this pull request? change since version in doc ## How was this patch tested? manual Author: Felix Cheung

spark git commit: [SPARK-17376][SPARKR] followup - change since version

2016-09-02 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master e79962f2f -> eac1d0e92 [SPARK-17376][SPARKR] followup - change since version ## What changes were proposed in this pull request? change since version in doc ## How was this patch tested? manual Author: Felix Cheung

spark git commit: [SPARK-16711] YarnShuffleService doesn't re-init properly on YARN rolling upgrade

2016-09-02 Thread vanzin
Repository: spark Updated Branches: refs/heads/master 419eefd81 -> e79962f2f [SPARK-16711] YarnShuffleService doesn't re-init properly on YARN rolling upgrade The Spark Yarn Shuffle Service doesn't re-initialize the application credentials early enough which causes any other spark executors

spark git commit: [SPARKR][DOC] regexp_extract should doc that it returns empty string when match fails

2016-09-02 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-2.0 29ac2f62e -> d4ae35d02 [SPARKR][DOC] regexp_extract should doc that it returns empty string when match fails ## What changes were proposed in this pull request? Doc change - see https://issues.apache.org/jira/browse/SPARK-16324 ##

spark git commit: [SPARKR][DOC] regexp_extract should doc that it returns empty string when match fails

2016-09-02 Thread shivaram
Repository: spark Updated Branches: refs/heads/master 812333e43 -> 419eefd81 [SPARKR][DOC] regexp_extract should doc that it returns empty string when match fails ## What changes were proposed in this pull request? Doc change - see https://issues.apache.org/jira/browse/SPARK-16324 ## How

spark git commit: [SPARK-17261] [PYSPARK] Using HiveContext after re-creating SparkContext in Spark 2.0 throws "Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext"

2016-09-02 Thread davies
Repository: spark Updated Branches: refs/heads/master 6bcbf9b74 -> ea6622865 [SPARK-17261] [PYSPARK] Using HiveContext after re-creating SparkContext in Spark 2.0 throws "Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext" ## What changes were proposed in this

spark git commit: [SPARK-17376][SPARKR] Spark version should be available in R

2016-09-02 Thread shivaram
Repository: spark Updated Branches: refs/heads/branch-2.0 30e5c8493 -> 29ac2f62e [SPARK-17376][SPARKR] Spark version should be available in R ## What changes were proposed in this pull request? Add sparkR.version() API. ``` > sparkR.version() [1] "2.1.0-SNAPSHOT" ``` ## How was this patch

spark git commit: [SPARK-17376][SPARKR] Spark version should be available in R

2016-09-02 Thread shivaram
Repository: spark Updated Branches: refs/heads/master ea6622865 -> 812333e43 [SPARK-17376][SPARKR] Spark version should be available in R ## What changes were proposed in this pull request? Add sparkR.version() API. ``` > sparkR.version() [1] "2.1.0-SNAPSHOT" ``` ## How was this patch

spark git commit: [SPARK-17261] [PYSPARK] Using HiveContext after re-creating SparkContext in Spark 2.0 throws "Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext"

2016-09-02 Thread davies
Repository: spark Updated Branches: refs/heads/branch-2.0 91a3cf136 -> 30e5c8493 [SPARK-17261] [PYSPARK] Using HiveContext after re-creating SparkContext in Spark 2.0 throws "Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext" ## What changes were proposed in

spark git commit: [SPARK-17351] Refactor JDBCRDD to expose ResultSet -> Seq[Row] utility methods

2016-09-02 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 806d8a8e9 -> 6bcbf9b74 [SPARK-17351] Refactor JDBCRDD to expose ResultSet -> Seq[Row] utility methods This patch refactors the internals of the JDBC data source in order to allow some of its code to be re-used in an automated comparison

spark git commit: [SPARK-16984][SQL] don't try whole dataset immediately when first partition doesn't have…

2016-09-02 Thread hvanhovell
Repository: spark Updated Branches: refs/heads/master 247a4faf0 -> 806d8a8e9 [SPARK-16984][SQL] don't try whole dataset immediately when first partition doesn't have… ## What changes were proposed in this pull request? Try increase number of partitions to try so we don't revert to all.

spark git commit: [SPARK-16935][SQL] Verification of Function-related ExternalCatalog APIs

2016-09-02 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.0 d9d10ffb9 -> 91a3cf136 [SPARK-16935][SQL] Verification of Function-related ExternalCatalog APIs Function-related `HiveExternalCatalog` APIs do not have enough verification logics. After the PR, `HiveExternalCatalog` and

spark git commit: [SPARK-16935][SQL] Verification of Function-related ExternalCatalog APIs

2016-09-02 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 7ee24dac8 -> 247a4faf0 [SPARK-16935][SQL] Verification of Function-related ExternalCatalog APIs ### What changes were proposed in this pull request? Function-related `HiveExternalCatalog` APIs do not have enough verification logics. After

spark git commit: [SPARK-17352][WEBUI] Executor computing time can be negative-number because of calculation error

2016-09-02 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.0 171bdfd96 -> d9d10ffb9 [SPARK-17352][WEBUI] Executor computing time can be negative-number because of calculation error ## What changes were proposed in this pull request? In StagePage, executor-computing-time is calculated but

spark git commit: [SPARK-17352][WEBUI] Executor computing time can be negative-number because of calculation error

2016-09-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master a3097e2b3 -> 7ee24dac8 [SPARK-17352][WEBUI] Executor computing time can be negative-number because of calculation error ## What changes were proposed in this pull request? In StagePage, executor-computing-time is calculated but

spark git commit: [SQL][DOC][MINOR] Add (Scala-specific) and (Java-specific)

2016-09-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master 6969dcc79 -> a3097e2b3 [SQL][DOC][MINOR] Add (Scala-specific) and (Java-specific) ## What changes were proposed in this pull request? Adds (Scala-specific) and (Java-specific) to Scaladoc. ## How was this patch tested? local build

spark git commit: [SPARK-15509][ML][SPARKR] R MLlib algorithms should support input columns "features" and "label"

2016-09-02 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master 0f30cdedb -> 6969dcc79 [SPARK-15509][ML][SPARKR] R MLlib algorithms should support input columns "features" and "label" https://issues.apache.org/jira/browse/SPARK-15509 ## What changes were proposed in this pull request? Currently in

spark git commit: [SPARK-16883][SPARKR] SQL decimal type is not properly cast to number when collecting SparkDataFrame

2016-09-02 Thread felixcheung
Repository: spark Updated Branches: refs/heads/branch-2.0 f9463238d -> 171bdfd96 [SPARK-16883][SPARKR] SQL decimal type is not properly cast to number when collecting SparkDataFrame ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix)

spark git commit: [SPARK-16883][SPARKR] SQL decimal type is not properly cast to number when collecting SparkDataFrame

2016-09-02 Thread felixcheung
Repository: spark Updated Branches: refs/heads/master 2ab8dbdda -> 0f30cdedb [SPARK-16883][SPARKR] SQL decimal type is not properly cast to number when collecting SparkDataFrame ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix)

spark git commit: [SPARK-17342][WEBUI] Style of event timeline is broken

2016-09-02 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.0 dd377a522 -> f9463238d [SPARK-17342][WEBUI] Style of event timeline is broken ## What changes were proposed in this pull request? SPARK-15373 (#13158) updated the version of vis.js to 4.16.1. As of 4.0.0, some class was renamed like

spark git commit: [SPARK-17342][WEBUI] Style of event timeline is broken

2016-09-02 Thread srowen
Repository: spark Updated Branches: refs/heads/master f2d6e2ef2 -> 2ab8dbdda [SPARK-17342][WEBUI] Style of event timeline is broken ## What changes were proposed in this pull request? SPARK-15373 (#13158) updated the version of vis.js to 4.16.1. As of 4.0.0, some class was renamed like