[GitHub] spark pull request #15532: [SPARK-17989][SQL] Check ascendingOrder type in s...

2016-10-18 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15532#discussion_r83930227 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -124,7 +124,14 @@ case class SortArray(base:

[GitHub] spark issue #15475: [SPARK-17991] [SQL] Enable metastore partition pruning b...

2016-10-18 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/15475 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15475: [SPARK-17991] [SQL] Enable metastore partition pruning b...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15475 **[Test build #67138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67138/consoleFull)** for PR 15475 at commit [`2581380`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15285 **[Test build #67133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67133/consoleFull)** for PR 15285 at commit [`b3041b7`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15285 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67133/ Test PASSed. ---

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15285 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15467: [SPARK-17912][SQL] Refactor code generation to get data ...

2016-10-18 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/15467 @ericl thank you very much --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/15285 LGTM. Merging to master and 2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wi

[GitHub] spark issue #14947: [SPARK-17388][SQL] Support for inferring type date/times...

2016-10-18 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/14947 Merging this into master, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark pull request #15522: [MINOR][DOC] Add more built-in sources in sql-pro...

2016-10-18 Thread weiqingy
Github user weiqingy commented on a diff in the pull request: https://github.com/apache/spark/pull/15522#discussion_r83941926 --- Diff: docs/sql-programming-guide.md --- @@ -422,8 +422,8 @@ In the simplest form, the default data source (`parquet` unless otherwise config You ca

[GitHub] spark pull request #14947: [SPARK-17388][SQL] Support for inferring type dat...

2016-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14947 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15285 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #13526: [SPARK-15780][SQL] Support mapValues on KeyValueGroupedD...

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/13526 cc @cloud-fan This looks good to me -- but I don't remember why we didn't merge it earlier already. --- If your project is set up for it, you can reply to this email and have your reply appe

[GitHub] spark issue #15496: [SPARK-17950] [Python] Match SparseVector behavior with ...

2016-10-18 Thread itg-abby
Github user itg-abby commented on the issue: https://github.com/apache/spark/pull/15496 It is possible that I am missing something or that I have unintentionally obfuscated this pull request, I will try summarizing my understanding/purpose and see if it sheds any light: Dense

[GitHub] spark issue #15530: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15530 **[Test build #67139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67139/consoleFull)** for PR 15530 at commit [`0257b9e`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #15533: [SPARK-17974] (try 2) Refactor FileCatalog classes to si...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15533 **[Test build #67136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67136/consoleFull)** for PR 15533 at commit [`699bce5`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15533: [SPARK-17974] (try 2) Refactor FileCatalog classes to si...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15533 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15533: [SPARK-17974] (try 2) Refactor FileCatalog classes to si...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15533 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67136/ Test PASSed. ---

[GitHub] spark issue #15534: [SPARK-17917][Scheduler] Fire SparkListenerTasksStarved ...

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15534 The user can track this pretty easily themselves, can't they? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15285 **[Test build #67135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67135/consoleFull)** for PR 15285 at commit [`1e3302d`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15533: [SPARK-17974] (try 2) Refactor FileCatalog classes to si...

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15533 Merging in master. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15525: [SPARK-17985][CORE] Bump commons-lang3 version to 3.5.

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15525 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15285 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15285 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67135/ Test PASSed. ---

[GitHub] spark pull request #15533: [SPARK-17974] (try 2) Refactor FileCatalog classe...

2016-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15533 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #15522: [MINOR][DOC] Add more built-in sources in sql-programmin...

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15522 Merging in master/branch-2.0. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #15525: [SPARK-17985][CORE] Bump commons-lang3 version to...

2016-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15525 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15285 **[Test build #67134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67134/consoleFull)** for PR 15285 at commit [`4ead779`](https://github.com/apache/spark/commit/

[GitHub] spark pull request #15522: [MINOR][DOC] Add more built-in sources in sql-pro...

2016-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15522 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15285 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67134/ Test PASSed. ---

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15285 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15514: [SPARK-17960][PySpark] [Upgrade to Py4J 0.10.4]

2016-10-18 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/15514 cc @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark pull request #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrai...

2016-10-18 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/15481#discussion_r83946120 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -392,9 +392,23 @@ class CoarseGrainedSchedulerBa

[GitHub] spark pull request #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrai...

2016-10-18 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/15481#discussion_r83946127 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -92,19 +92,19 @@ class CoarseGrainedSchedulerBac

[GitHub] spark issue #15512: [SPARK-17930][CORE]The SerializerInstance instance used ...

2016-10-18 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15512 LGTM. Merging to master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark issue #15499: [SPARK-17955][SQL] Make DataFrameReader.jdbc call DataFr...

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15499 Merging in master. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactore...

2016-10-18 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/15535 [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored StreamingQueryListener APIs for branch-2.0 This is the branch-2.0 PR of #15530 to make the APIs consistent with the master. Since these APIs are exp

[GitHub] spark pull request #15512: [SPARK-17930][CORE]The SerializerInstance instanc...

2016-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15512 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark pull request #15499: [SPARK-17955][SQL] Make DataFrameReader.jdbc call...

2016-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15499 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #15529: [SPARK-17751] [SQL] [Backport-2.0] Remove spark.sql.eage...

2016-10-18 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15529 Thanks! Close it now --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #15529: [SPARK-17751] [SQL] [Backport-2.0] Remove spark.s...

2016-10-18 Thread gatorsmile
Github user gatorsmile closed the pull request at: https://github.com/apache/spark/pull/15529 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15535 **[Test build #67140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67140/consoleFull)** for PR 15535 at commit [`0fe56d2`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #15521: [SPARK-17980] [SQL] Fix refreshByPath for converted Hive...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15521 **[Test build #67137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67137/consoleFull)** for PR 15521 at commit [`5088ae0`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15521: [SPARK-17980] [SQL] Fix refreshByPath for converted Hive...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15521 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67137/ Test PASSed. ---

[GitHub] spark issue #15521: [SPARK-17980] [SQL] Fix refreshByPath for converted Hive...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15521 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15450: [SPARK-3261] [MLLIB] KMeans clusterer can return duplica...

2016-10-18 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/15450 I don't feel strongly either way, but I don't like the potential of this: scala model.getK scala> 3 model.clusterCenters.length scala> 1 Should we conside

[GitHub] spark issue #15530: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/15530 @rxin Can you take a look once again. Note that there is a separate PR for the same changes in branch-2.0 (with different Mima excludes) - https://github.com/apache/spark/pull/15535/files --- If y

[GitHub] spark issue #15525: [SPARK-17985][CORE] Bump commons-lang3 version to 3.5.

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15525 Actually I'm seeing the following exceptions locally as well as on Jenkins for Hadoop 2.2: ``` [error] /scratch/rxin/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:1489: valu

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15535 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15535 **[Test build #67141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67141/consoleFull)** for PR 15535 at commit [`9549e58`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #15526: [SPARK-17986] [ML] SQLTransformer should remove temporar...

2016-10-18 Thread drewrobb
Github user drewrobb commented on the issue: https://github.com/apache/spark/pull/15526 @yanboliang would you be able to review this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark issue #15530: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15530 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15407 Thanks for the explanation. Merging in master/branch-2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #15490: [SPARK-10541] [Web UI] Allow ApplicationHistoryPr...

2016-10-18 Thread ajbozarth
Github user ajbozarth commented on a diff in the pull request: https://github.com/apache/spark/pull/15490#discussion_r83950235 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -262,6 +263,17 @@ private[history] class FsHistoryProvider(con

[GitHub] spark pull request #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue

2016-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15407 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #15407: [SPARK-17841][STREAMING][KAFKA] drain commitQueue

2016-10-18 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15407 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featu

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15535 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15535 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67140/ Test FAILed. ---

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15535 **[Test build #67140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67140/consoleFull)** for PR 15535 at commit [`0fe56d2`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSche...

2016-10-18 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15481 @scwf I think the initial fix with a small change might be sufficient. What I meant was something like this : ``` protected def reset(): Unit = { val executors = synchroniz

[GitHub] spark pull request #15536: [SPARK-13275] [Web UI] Visually clarified executo...

2016-10-18 Thread ajbozarth
GitHub user ajbozarth opened a pull request: https://github.com/apache/spark/pull/15536 [SPARK-13275] [Web UI] Visually clarified executors start time in timeline ## What changes were proposed in this pull request? Updated the Executors added/removed bubble in the time line

[GitHub] spark issue #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSche...

2016-10-18 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15481 > reset is called in YarnSchedulerEndpoint which ideally should not be a blocking action. I'm wondering if we can also fix this. --- If your project is set up for it, you can reply to this

[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns

2016-10-18 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/15274 @DanielMe oh, I see. `get_json_object` will not parse json array. You need to have a UDF to do that for Spark 1.6. --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] spark issue #15536: [SPARK-13275] [Web UI] Visually clarified executors star...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15536 **[Test build #67142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67142/consoleFull)** for PR 15536 at commit [`bf0117b`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #15536: [SPARK-13275] [Web UI] Visually clarified executors star...

2016-10-18 Thread ajbozarth
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/15536 This is a purely aesthetic update as the lines already show the exact time, but given jobs are shown as ranges the executor bubbles center alignment on the line was confusing visually for some.

[GitHub] spark issue #15398: [SPARK-17647][SQL][WIP] Fix backslash escaping in 'LIKE'...

2016-10-18 Thread jodersky
Github user jodersky commented on the issue: https://github.com/apache/spark/pull/15398 Agreed, I'll update the regex parsing to copy Hive's behaviour --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark issue #15481: [SPARK-17929] [CORE] Fix deadlock when CoarseGrainedSche...

2016-10-18 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/15481 @zsxwing Ah, then simply making it send() instead of askWithRetry() should do, no ? That was actually in the initial PR - I was not sure if we want to change the behavior from askWithRetry to se

[GitHub] spark issue #15445: [SPARK-17817][PySpark][FOLLOWUP] PySpark RDD Repartition...

2016-10-18 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/15445 @viirya That's good point, javaToPython can only used for known types that could deserialized in Java (for example, some types in sql and ml), this PR make better sense. Merging this into ma

[GitHub] spark pull request #15445: [SPARK-17817][PySpark][FOLLOWUP] PySpark RDD Repa...

2016-10-18 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15445 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #10154: [SPARK-12149] [Web UI] Executor UI improvement suggestio...

2016-10-18 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/10154 So I looked at the UI today and I have to say the color code is *extremely confusing*. On a normal cluster I saw a bunch of reds, which is usually reserved for errors. (I'm going to leave

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15535 **[Test build #67143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67143/consoleFull)** for PR 15535 at commit [`bb1b234`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #10154: [SPARK-12149] [Web UI] Executor UI improvement suggestio...

2016-10-18 Thread ajbozarth
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/10154 @rxin The reds only show up on failed tasks or a high gc time, what would you expect from each color? Also this was a while ago (and one of my first prs) so I'm willing to revisit how this was don

[GitHub] spark issue #15285: [SPARK-17711] Compress rolled executor log

2016-10-18 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15285 This one breaks hadoop-2.2 builds: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.2/1951/console --- If your project is set up for it, you can reply to this email and hav

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15535 **[Test build #67144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67144/consoleFull)** for PR 15535 at commit [`4cac044`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15535 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15535 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67143/ Test FAILed. ---

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15535 **[Test build #67143 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67143/consoleFull)** for PR 15535 at commit [`bb1b234`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15535 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67141/ Test FAILed. ---

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15535 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/15535 I cancelled previous jenkins tests, hence the failures. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark issue #15475: [SPARK-17991] [SQL] Enable metastore partition pruning b...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15475 **[Test build #67138 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67138/consoleFull)** for PR 15475 at commit [`2581380`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15475: [SPARK-17991] [SQL] Enable metastore partition pruning b...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15475 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #15535: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15535 **[Test build #3361 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3361/consoleFull)** for PR 15535 at commit [`4cac044`](https://github.com/apache/spark/commit/

[GitHub] spark issue #15475: [SPARK-17991] [SQL] Enable metastore partition pruning b...

2016-10-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15475 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67138/ Test PASSed. ---

[GitHub] spark pull request #15218: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-18 Thread wangmiao1981
Github user wangmiao1981 commented on a diff in the pull request: https://github.com/apache/spark/pull/15218#discussion_r83961053 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala --- @@ -0,0 +1,154 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] spark issue #15530: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15530 **[Test build #67145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67145/consoleFull)** for PR 15530 at commit [`1679077`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #15421: [SPARK-17811] SparkR cannot parallelize data.frame with ...

2016-10-18 Thread falaki
Github user falaki commented on the issue: https://github.com/apache/spark/pull/15421 @wangmiao1981 would you please also test the master branch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark pull request #15537: [SPARK-17711] [test-hadoop2.2] Fix hadoop2.2 comp...

2016-10-18 Thread loneknightpy
GitHub user loneknightpy opened a pull request: https://github.com/apache/spark/pull/15537 [SPARK-17711] [test-hadoop2.2] Fix hadoop2.2 compilation error ## What changes were proposed in this pull request? Fix hadoop2.2 compilation error. ## How was this patch teste

[GitHub] spark pull request #15218: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-18 Thread wangmiao1981
Github user wangmiao1981 commented on a diff in the pull request: https://github.com/apache/spark/pull/15218#discussion_r83962588 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskAssigner.scala --- @@ -0,0 +1,151 @@ +/* + * Licensed to the Apache Software Founda

[GitHub] spark issue #15537: [SPARK-17711] [test-hadoop2.2] Fix hadoop2.2 compilation...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15537 **[Test build #67146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67146/consoleFull)** for PR 15537 at commit [`fd0b076`](https://github.com/apache/spark/commit/f

[GitHub] spark pull request #15218: [SPARK-17637][Scheduler]Packed scheduling for Spa...

2016-10-18 Thread wangmiao1981
Github user wangmiao1981 commented on a diff in the pull request: https://github.com/apache/spark/pull/15218#discussion_r83963294 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -250,24 +263,26 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark issue #15496: [SPARK-17950] [Python] Match SparseVector behavior with ...

2016-10-18 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/15496 @itg-abby I see what you're going for, but I don't think it's a great idea in general. For many operations on sparse vectors, we should not materialize the sparse vector as dense. I'm not sure if it'

[GitHub] spark issue #15537: [SPARK-17711] [test-hadoop2.2] Fix hadoop2.2 compilation...

2016-10-18 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15537 LGTM pending tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark pull request #14553: [SPARK-16963] [STREAMING] [SQL] Changes to Source...

2016-10-18 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/14553#discussion_r83953473 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Source.scala --- @@ -30,16 +30,30 @@ trait Source { /** Returns the schem

[GitHub] spark pull request #14553: [SPARK-16963] [STREAMING] [SQL] Changes to Source...

2016-10-18 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/14553#discussion_r83965312 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala --- @@ -336,17 +342,27 @@ class StreamExecution( }

[GitHub] spark pull request #14553: [SPARK-16963] [STREAMING] [SQL] Changes to Source...

2016-10-18 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/14553#discussion_r83965244 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala --- @@ -336,17 +342,27 @@ class StreamExecution( }

[GitHub] spark issue #15398: [SPARK-17647][SQL] Fix backslash escaping in 'LIKE' patt...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15398 **[Test build #67147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67147/consoleFull)** for PR 15398 at commit [`85a3a38`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #15490: [SPARK-10541] [Web UI] Allow ApplicationHistoryProviders...

2016-10-18 Thread ajbozarth
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/15490 The dir is actually already displayed so I just added a few words to clarify that the current setting is displayed above. ![screen shot 2016-10-18 at 3 48 39 pm](https://cloud.githubuserc

[GitHub] spark issue #15532: [SPARK-17989][SQL] Check ascendingOrder type in sort_arr...

2016-10-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15532 Thank you both. Will adress the comments soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #15530: [SPARK-17731][SQL][STREAMING][FOLLOWUP] Refactored Strea...

2016-10-18 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15530 **[Test build #67139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67139/consoleFull)** for PR 15530 at commit [`0257b9e`](https://github.com/apache/spark/commit/

<    1   2   3   4   5   6   >