[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r103337724 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2512,3 +2522,67 @@ object ResolveCreateNamedStruct

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r102165726 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -707,13 +709,85 @@ class Analyzer( }

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r102256790 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2332,6 +2337,11 @@ class Analyzer(

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r103340692 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala --- @@ -40,19 +42,179 @@ abstract class PlanExpression[T

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r103340272 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -123,19 +123,36 @@ case class Not(child:

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r103339031 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -109,6 +109,26 @@ object TypeCoercion { }

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r102167746 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1110,31 +1184,24 @@ class Analyzer( }

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r103336411 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1398,42 +1399,46 @@ class Analyzer( }

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r102168672 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1110,31 +1184,24 @@ class Analyzer( }

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r102168299 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -707,13 +709,85 @@ class Analyzer( }

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r102167233 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1110,31 +1184,24 @@ class Analyzer( }

[GitHub] spark pull request #16954: [SPARK-18874][SQL] First phase: Deferring the cor...

2017-02-27 Thread hvanhovell
Github user hvanhovell commented on a diff in the pull request: https://github.com/apache/spark/pull/16954#discussion_r102168200 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1110,31 +1184,24 @@ class Analyzer( }

[GitHub] spark pull request #17078: [SPARK-19746][ML] Faster indexing for logistic ag...

2017-02-27 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/17078#discussion_r103342591 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala --- @@ -456,6 +456,32 @@ class LogisticRegressionSuite

[GitHub] spark pull request #17078: [SPARK-19746][ML] Faster indexing for logistic ag...

2017-02-27 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/17078#discussion_r103342093 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -1447,7 +1447,7 @@ private class LogisticAggregator(

[GitHub] spark pull request #17078: [SPARK-19746][ML] Faster indexing for logistic ag...

2017-02-27 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/17078#discussion_r103342317 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala --- @@ -1431,7 +1431,12 @@ private class LogisticAggregator(

[GitHub] spark issue #16965: [SPARK-18450][ML] Scala API Change for LSH AND-amplifica...

2017-02-27 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16965 Github isn't handling the merge well, so you might try rebasing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #17015: [SPARK-19678][SQL] remove MetastoreRelation

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17015 **[Test build #73536 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73536/testReport)** for PR 17015 at commit

[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...

2017-02-27 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/16819 I agree with others, this is not the way to do this. There are different schedulers in yarn, each with different configs that could affect the actual resources you get. If you want to

[GitHub] spark issue #14273: [SPARK-9140] [ML] Replace TimeTracker by MultiStopwatch

2017-02-27 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/14273 Sorry about the delay here. Do you still have time to work on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #16929: [SPARK-19595][SQL] Support json array in from_jso...

2017-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16929#discussion_r103338371 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -480,36 +480,79 @@ case class

[GitHub] spark issue #16929: [SPARK-19595][SQL] Support json array in from_json

2017-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16929 Thanks for your detailed look. Let me check again and address the comments! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request #16811: [SPARK-17629][ML] methods to return synonyms dire...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16811#discussion_r103338261 --- Diff: mllib/src/test/scala/org/apache/spark/ml/feature/Word2VecSuite.scala --- @@ -144,6 +144,31 @@ class Word2VecSuite extends SparkFunSuite with

[GitHub] spark pull request #16811: [SPARK-17629][ML] methods to return synonyms dire...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16811#discussion_r103338146 --- Diff: mllib/src/test/scala/org/apache/spark/ml/feature/Word2VecSuite.scala --- @@ -144,6 +144,31 @@ class Word2VecSuite extends SparkFunSuite with

[GitHub] spark pull request #16929: [SPARK-19595][SQL] Support json array in from_jso...

2017-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16929#discussion_r103337914 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -480,36 +480,79 @@ case class

[GitHub] spark pull request #16929: [SPARK-19595][SQL] Support json array in from_jso...

2017-02-27 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/16929#discussion_r103337028 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2969,11 +2969,27 @@ object functions { } /** - *

[GitHub] spark pull request #16929: [SPARK-19595][SQL] Support json array in from_jso...

2017-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16929#discussion_r103334238 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -480,36 +480,79 @@ case class

[GitHub] spark pull request #16929: [SPARK-19595][SQL] Support json array in from_jso...

2017-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16929#discussion_r10990 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2969,11 +2969,27 @@ object functions { } /** -

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103325403 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -34,8 +36,25 @@ import org.apache.spark.util.collection.OpenHashMap

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103330093 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -34,8 +36,25 @@ import org.apache.spark.util.collection.OpenHashMap

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103332623 --- Diff: mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala --- @@ -75,22 +75,32 @@ class StringIndexerSuite

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103332929 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -34,8 +36,25 @@ import org.apache.spark.util.collection.OpenHashMap

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103325211 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -17,14 +17,16 @@ package org.apache.spark.ml.feature

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103331212 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -163,25 +190,28 @@ class StringIndexerModel ( }

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103330268 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -71,18 +90,22 @@ class StringIndexer @Since("1.4.0") ( def

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103330303 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -71,18 +90,22 @@ class StringIndexer @Since("1.4.0") ( def

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103330242 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -71,18 +90,22 @@ class StringIndexer @Since("1.4.0") ( def

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103331444 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -163,25 +190,28 @@ class StringIndexerModel ( }

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103332885 --- Diff: docs/ml-features.md --- @@ -576,7 +578,22 @@ will be generated: 2 | c| 1.0 -Notice that the row containing

[GitHub] spark pull request #16883: [SPARK-17498][ML] StringIndexer enhancement for h...

2017-02-27 Thread jkbradley
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/16883#discussion_r103332764 --- Diff: docs/ml-features.md --- @@ -502,7 +502,7 @@ for more details on the API. ## StringIndexer `StringIndexer` encodes a string

[GitHub] spark issue #17085: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-02-27 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/17085 @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley @actuaryzhang @srowen would you be able to take a look? I've split the larger pull request into three parts as suggested. --- If your

[GitHub] spark issue #17086: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-02-27 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/17086 @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley @actuaryzhang @srowen would you be able to take a look? I've split the larger pull request into three parts as suggested. --- If your

[GitHub] spark issue #17084: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-02-27 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/17084 @sethah @Lewuathe @thunterdb @WeichenXu123 @jkbradley @actuaryzhang @srowen would you be able to take a look? I've split the larger pull request into three parts as suggested. --- If your

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread mgummelt
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17031 It depends on the application. It's the amount of time you have to wait before having the opportunity to use those resources again. But if you explicitly revive, which we do here whenever we

[GitHub] spark pull request #17071: [SPARK-15615][SQL][BUILD][FOLLOW-UP] Replace depr...

2017-02-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17071 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17071: [SPARK-15615][SQL][BUILD][FOLLOW-UP] Replace deprecated ...

2017-02-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17071 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17075: [SPARK-19727][SQL] Fix for round function that modifies ...

2017-02-27 Thread wojtek-szymanski
Github user wojtek-szymanski commented on the issue: https://github.com/apache/spark/pull/17075 Good idea @cloud-fan. I will look for usages of `changePrecision` then. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #17077: [SPARK-16931][PYTHON][SQL] Add Python wrapper for bucket...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17077 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17077: [SPARK-16931][PYTHON][SQL] Add Python wrapper for bucket...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17077 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73535/ Test PASSed. ---

[GitHub] spark issue #17077: [SPARK-16931][PYTHON][SQL] Add Python wrapper for bucket...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17077 **[Test build #73535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73535/testReport)** for PR 17077 at commit

[GitHub] spark issue #15628: [SPARK-17471][ML] Add compressed method to ML matrices

2017-02-27 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/15628 re-ping @dbtsai @MLnick @yanboliang I still think this is an important patch :D --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #17077: [SPARK-16931][PYTHON][SQL] Add Python wrapper for bucket...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17077 **[Test build #73535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73535/testReport)** for PR 17077 at commit

[GitHub] spark issue #16883: [SPARK-17498][ML] StringIndexer enhancement for handling...

2017-02-27 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16883 I'll take a look now, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-02-27 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/16867 This looks like a real test failure resulting from this change --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #17088: [SPARK-19753][CORE] All shuffle files on a host should b...

2017-02-27 Thread sitalkedia
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17088 >> fetch failure does not imply lost executor - it could be a transient issue. Similarly, executor loss does not imply host loss. You are right, it could be transient, but we do have

[GitHub] spark issue #17087: [SPARK-19372][SQL] Fix throwing a Java exception at df.f...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17087 **[Test build #73530 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73530/testReport)** for PR 17087 at commit

[GitHub] spark issue #17088: [SPARK-19753][CORE] All shuffle files on a host should b...

2017-02-27 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17088 fetch failure does not imply lost executor - it could be a transient issue. Similarly, executor loss does not imply host loss. This is quite drastic for a fetch failure : spark already

[GitHub] spark issue #17085: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17085 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73528/ Test PASSed. ---

[GitHub] spark issue #17085: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17085 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #13932: [SPARK-15354] [CORE] Topology aware block replication st...

2017-02-27 Thread shubhamchopra
Github user shubhamchopra commented on the issue: https://github.com/apache/spark/pull/13932 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #13932: [SPARK-15354] [CORE] Topology aware block replication st...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13932 **[Test build #73534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73534/testReport)** for PR 13932 at commit

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/17031 Ok I see. LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #13143: [SPARK-15359] [Mesos] Mesos dispatcher should handle DRI...

2017-02-27 Thread mgummelt
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/13143 What whole function is designed poorly. We need to totally change it instead of tacking this on. We shouldn't be calling `driver.run()` in a separate thread. We should be calling

[GitHub] spark issue #13932: [SPARK-15354] [CORE] Topology aware block replication st...

2017-02-27 Thread shubhamchopra
Github user shubhamchopra commented on the issue: https://github.com/apache/spark/pull/13932 Rebased to resolve merge conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17085: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17085 **[Test build #73528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73528/testReport)** for PR 17085 at commit

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16867 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16867 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73514/ Test FAILed. ---

[GitHub] spark issue #13326: [SPARK-15560] [Mesos] Queued/Supervise drivers waiting f...

2017-02-27 Thread mgummelt
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/13326 A killed driver never finished, so it shouldn't be added to the finished set. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #16867: [SPARK-16929] Improve performance when check speculatabl...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16867 **[Test build #73514 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73514/testReport)** for PR 16867 at commit

[GitHub] spark issue #17088: [SPARK-19753][CORE] All shuffle files on a host should b...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17088 **[Test build #73533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73533/testReport)** for PR 17088 at commit

[GitHub] spark issue #16959: [SPARK-19631][CORE] OutputCommitCoordinator should not a...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16959 **[Test build #73532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73532/testReport)** for PR 16959 at commit

[GitHub] spark pull request #17088: [SPARK-19753][CORE] All shuffle files on a host s...

2017-02-27 Thread sitalkedia
GitHub user sitalkedia opened a pull request: https://github.com/apache/spark/pull/17088 [SPARK-19753][CORE] All shuffle files on a host should be removed in … ## What changes were proposed in this pull request? Currently, when we detect fetch failure, we only remove the

[GitHub] spark issue #16959: [SPARK-19631][CORE] OutputCommitCoordinator should not a...

2017-02-27 Thread pwoody
Github user pwoody commented on the issue: https://github.com/apache/spark/pull/16959 Thanks for the feedback @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17059: [SPARK-19733][ML]Removed unnecessary castings and refact...

2017-02-27 Thread datumbox
Github user datumbox commented on the issue: https://github.com/apache/spark/pull/17059 @srowen @mlnick I updated the PR based on what was discussed above and I tested it again on Spark 2.1. I also updated the coding styles and the exception message. The changes requested

[GitHub] spark issue #16557: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16557 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16557: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16557 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73525/ Test PASSed. ---

[GitHub] spark issue #16557: [SPARK-18693][ML][MLLIB] ML Evaluators should use weight...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16557 **[Test build #73525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73525/testReport)** for PR 16557 at commit

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17031 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17031 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73531/ Test PASSed. ---

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17031 **[Test build #73531 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73531/testReport)** for PR 17031 at commit

[GitHub] spark pull request #17031: [SPARK-19702][MESOS] Add suppress/revive support ...

2017-02-27 Thread mgummelt
Github user mgummelt commented on a diff in the pull request: https://github.com/apache/spark/pull/17031#discussion_r103303266 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -582,141 +688,33 @@

[GitHub] spark pull request #17031: [SPARK-19702][MESOS] Add suppress/revive support ...

2017-02-27 Thread mgummelt
Github user mgummelt commented on a diff in the pull request: https://github.com/apache/spark/pull/17031#discussion_r103303283 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -737,13 +735,75 @@

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread mgummelt
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17031 @skonto @susanxhuynh I've updated the solution to use a longer (120s) default refuse timeout, instead of suppressing offers. Please re-review. Just as the previous refuse seconds settings were

[GitHub] spark pull request #17048: [SPARK-14772][PYTHON][ML] Fixed Params.copy metho...

2017-02-27 Thread BryanCutler
Github user BryanCutler closed the pull request at: https://github.com/apache/spark/pull/17048 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17031 **[Test build #73531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73531/testReport)** for PR 17031 at commit

[GitHub] spark issue #17031: [SPARK-19702][MESOS] Add suppress/revive support to the ...

2017-02-27 Thread mgummelt
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17031 @skonto Cassandra supports suppress/revive

[GitHub] spark pull request #16929: [SPARK-19595][SQL] Support json array in from_jso...

2017-02-27 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/16929#discussion_r103302035 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -480,36 +480,79 @@ case class

[GitHub] spark pull request #16929: [SPARK-19595][SQL] Support json array in from_jso...

2017-02-27 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/16929#discussion_r103300622 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2969,11 +2969,27 @@ object functions { } /** - *

[GitHub] spark issue #15770: [SPARK-15784][ML]:Add Power Iteration Clustering to spar...

2017-02-27 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/15770 Yep, that's correct. Everyone, please let me know if you disagree. Also, if we do go with Option 2 above, then the input schema could be a few possible things: * list of (neighbor

[GitHub] spark issue #17048: [SPARK-14772][PYTHON][ML] Fixed Params.copy method to ma...

2017-02-27 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/17048 Can you please close this manually? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16782: [SPARK-19348][PYTHON][WIP] PySpark keyword_only decorato...

2017-02-27 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16782 I'm OK with the current solution, though if it's easy to check using ```inspection``` then that seems nice to do. If there are cases in which the wrapper is still not thread-safe, then

[GitHub] spark pull request #16959: [SPARK-19631][CORE] OutputCommitCoordinator shoul...

2017-02-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16959#discussion_r103296961 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -181,11 +194,19 @@ private[spark] class

[GitHub] spark pull request #16959: [SPARK-19631][CORE] OutputCommitCoordinator shoul...

2017-02-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16959#discussion_r103296737 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -137,10 +141,15 @@ private[spark] class

[GitHub] spark pull request #16959: [SPARK-19631][CORE] OutputCommitCoordinator shoul...

2017-02-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16959#discussion_r103295542 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -48,25 +48,28 @@ private[spark] class

[GitHub] spark pull request #16959: [SPARK-19631][CORE] OutputCommitCoordinator shoul...

2017-02-27 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/16959#discussion_r103297378 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -158,13 +167,17 @@ private[spark] class

[GitHub] spark issue #17087: [SPARK-19372][SQL] Fix throwing a Java exception at df.f...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17087 **[Test build #73530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73530/testReport)** for PR 17087 at commit

[GitHub] spark pull request #17087: [SPARK-19372][SQL] Fix throwing a Java exception ...

2017-02-27 Thread kiszk
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/17087 [SPARK-19372][SQL] Fix throwing a Java exception at df.fliter() due to 64KB bytecode size limit ## What changes were proposed in this pull request? When an expression for `df.filter()` has

[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73515/ Test PASSed. ---

[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...

2017-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...

2017-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73515/testReport)** for PR 16819 at commit

[GitHub] spark pull request #16639: [SPARK-19276][CORE] Fetch Failure handling robust...

2017-02-27 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16639#discussion_r103289664 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -400,8 +410,16 @@ private[spark] class Executor(

<    1   2   3   4   5   6   7   >