[GitHub] spark pull request #12646: [SPARK-14878][SQL] Trim characters string functio...

2017-05-03 Thread kevinyu98
Github user kevinyu98 commented on a diff in the pull request: https://github.com/apache/spark/pull/12646#discussion_r114703725 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -461,68 +462,270 @@ case class

[GitHub] spark issue #16561: [SPARK-18801][SQL][FOLLOWUP] Alias the view with its chi...

2017-05-03 Thread QQshu1
Github user QQshu1 commented on the issue: https://github.com/apache/spark/pull/16561 hi , I have a question, why we should Eliminate View in the first of the optimizer.? thank you.@jiangxb1987 --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread zero323
Github user zero323 commented on a diff in the pull request: https://github.com/apache/spark/pull/17851#discussion_r114702184 --- Diff: R/pkg/R/DataFrame.R --- @@ -3715,3 +3715,34 @@ setMethod("rollup", sgd <- callJMethod(x@sdf, "rollup", jcol)

[GitHub] spark pull request #17678: [SPARK-20381][SQL] Add SQL metrics of numOutputRo...

2017-05-03 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/17678#discussion_r114700907 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala --- @@ -83,6 +85,7 @@ class

[GitHub] spark issue #17814: [SPARK-20015][SPARKR][SS][DOC][EXAMPLE] Document R Struc...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17814 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76439/ Test PASSed. ---

[GitHub] spark issue #17814: [SPARK-20015][SPARKR][SS][DOC][EXAMPLE] Document R Struc...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17814 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17814: [SPARK-20015][SPARKR][SS][DOC][EXAMPLE] Document R Struc...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17814 **[Test build #76439 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76439/testReport)** for PR 17814 at commit

[GitHub] spark issue #17856: [SPARK-19660][SQL] Replace the deprecated property name ...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17856 **[Test build #76441 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76441/testReport)** for PR 17856 at commit

[GitHub] spark pull request #17856: [SPARK-19660][SQL] Replace the deprecated propert...

2017-05-03 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/17856 [SPARK-19660][SQL] Replace the deprecated property name fs.default.name to fs.defaultFS that newly introduced ## What changes were proposed in this pull request? Replace the deprecated

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

2017-05-03 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17825 you know - it would definitely be a better experience for the R user, so we should try that - it might break with the generic in `stats::alias` though and speaking of, we should

[GitHub] spark issue #17817: [SPARK-20543][SPARKR] skip tests when running on CRAN

2017-05-03 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17817 merged to master/2.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17750: [SPARK-4899][MESOS] Support for checkpointing on Coarse ...

2017-05-03 Thread lins05
Github user lins05 commented on the issue: https://github.com/apache/spark/pull/17750 IMO we should not enable checkpointing in fine-grained mode. Because with checkpointing enabled, mesos agents would persist all status updates to disk which means great I/O cost because fine-grained

[GitHub] spark issue #17855: [INFRA] Close stale PRs

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17855 **[Test build #76440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76440/testReport)** for PR 17855 at commit

[GitHub] spark issue #17855: [INFRA] Close stale PRs

2017-05-03 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17855 cc @srowen, could you take a look please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17855: [INFRA] Close stale PRs

2017-05-03 Thread HyukjinKwon
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/17855 [INFRA] Close stale PRs ## What changes were proposed in this pull request? This PR proposes to close a stale PR, several PRs suggested to be closed by a committer and obviously

[GitHub] spark pull request #17817: [SPARK-20543][SPARKR] skip tests when running on ...

2017-05-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17817 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17814: [SPARK-20015][SPARKR][SS][DOC][EXAMPLE] Document R Struc...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17814 **[Test build #76439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76439/testReport)** for PR 17814 at commit

[GitHub] spark issue #11119: [SPARK-10780][ML] Add an initial model to kmeans

2017-05-03 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/9 Do you guys mind if I propose to close this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17854: [SPARK-20564][Deploy] Reduce massive executor failures w...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17854 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17854: [SPARK-20564][Deploy] Reduce massive executor fai...

2017-05-03 Thread mariahualiu
GitHub user mariahualiu opened a pull request: https://github.com/apache/spark/pull/17854 [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) ## What changes were proposed in this pull request? In applications that use over 2000

[GitHub] spark pull request #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17851#discussion_r114698221 --- Diff: R/pkg/R/DataFrame.R --- @@ -3715,3 +3715,34 @@ setMethod("rollup", sgd <- callJMethod(x@sdf, "rollup", jcol)

[GitHub] spark pull request #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17851#discussion_r114698264 --- Diff: R/pkg/R/DataFrame.R --- @@ -3715,3 +3715,34 @@ setMethod("rollup", sgd <- callJMethod(x@sdf, "rollup", jcol)

[GitHub] spark pull request #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17851#discussion_r114698385 --- Diff: R/pkg/R/DataFrame.R --- @@ -3715,3 +3715,34 @@ setMethod("rollup", sgd <- callJMethod(x@sdf, "rollup", jcol)

[GitHub] spark pull request #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17851#discussion_r114698072 --- Diff: R/pkg/R/DataFrame.R --- @@ -3715,3 +3715,34 @@ setMethod("rollup", sgd <- callJMethod(x@sdf, "rollup", jcol)

[GitHub] spark pull request #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17851#discussion_r114698515 --- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R --- @@ -2147,6 +2147,18 @@ test_that("join(), crossJoin() and merge() on a DataFrame", {

[GitHub] spark pull request #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17851#discussion_r114698548 --- Diff: R/pkg/R/generics.R --- @@ -572,6 +572,10 @@ setGeneric("first", function(x, ...) { standardGeneric("first") }) #' @export

[GitHub] spark pull request #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17851#discussion_r114698295 --- Diff: R/pkg/R/DataFrame.R --- @@ -3715,3 +3715,34 @@ setMethod("rollup", sgd <- callJMethod(x@sdf, "rollup", jcol)

[GitHub] spark issue #17853: Branch 2.2

2017-05-03 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17853 @crjk21 it looks mistakenly open. Could you close this please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #17853: Branch 2.2

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17853 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17853: Branch 2.2

2017-05-03 Thread crjk21
GitHub user crjk21 opened a pull request: https://github.com/apache/spark/pull/17853 Branch 2.2 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch

[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register

2017-05-03 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/17831 Thanks for making this PR with the details @gatorsmile it appears to be orthogonal to this change. Historically we've treated Python API parity fixes as closer to bug fixes rather than new features

[GitHub] spark issue #17836: [SPARK-20566][SQL] ColumnVector should support `appendFl...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17836 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76438/ Test PASSed. ---

[GitHub] spark issue #17836: [SPARK-20566][SQL] ColumnVector should support `appendFl...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17836 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2017-05-03 Thread jinxing64
Github user jinxing64 commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r114696480 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java --- @@ -100,7 +114,14 @@ public void

[GitHub] spark issue #17836: [SPARK-20566][SQL] ColumnVector should support `appendFl...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17836 **[Test build #76438 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76438/testReport)** for PR 17836 at commit

[GitHub] spark pull request #17850: [SPARK-20584][PYSPARK][SQL] Python generic hint s...

2017-05-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17850 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17850: [SPARK-20584][PYSPARK][SQL] Python generic hint support

2017-05-03 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17850 Merging in master/2.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76434/ Test PASSed. ---

[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17100 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register

2017-05-03 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17831 cc @viirya too who I believe is appropriate to review this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17100 **[Test build #76434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76434/testReport)** for PR 17100 at commit

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17825 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76437/ Test PASSed. ---

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17825 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17825 **[Test build #76437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76437/testReport)** for PR 17825 at commit

[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...

2017-05-03 Thread guoxiaolongzte
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17841 Where is the risk? High concurrency? Transaction processing? This is where I am puzzled. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #17836: [SPARK-20566][SQL] ColumnVector should support `appendFl...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17836 **[Test build #76438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76438/testReport)** for PR 17836 at commit

[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...

2017-05-03 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 To ask a more direct question: The only public interface being added in this change is `ServiceCredentialProvider`. It's an interface that service-specific libraries (e.g. a Solr connector,

[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register

2017-05-03 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17831 It sounds orthogonal to me as well. LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17540 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17540 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76431/ Test PASSed. ---

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17540 **[Test build #76431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76431/testReport)** for PR 17540 at commit

[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...

2017-05-03 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17841 I think @srowen already clarified it very clearly, you can use it at your own risk, but to make it public and add to the doc should be well considered. --- If your project is set up for it, you

[GitHub] spark pull request #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

2017-05-03 Thread zero323
Github user zero323 commented on a diff in the pull request: https://github.com/apache/spark/pull/17825#discussion_r114687096 --- Diff: R/pkg/R/DataFrame.R --- @@ -3715,3 +3715,25 @@ setMethod("rollup", sgd <- callJMethod(x@sdf, "rollup", jcol)

[GitHub] spark issue #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76436/ Test PASSed. ---

[GitHub] spark issue #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17851 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17851 **[Test build #76436 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76436/testReport)** for PR 17851 at commit

[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...

2017-05-03 Thread guoxiaolongzte
Github user guoxiaolongzte commented on the issue: https://github.com/apache/spark/pull/17841 @jerryshao I said that the use of the scene is real, do you agree? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17825 **[Test build #76437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76437/testReport)** for PR 17825 at commit

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

2017-05-03 Thread zero323
Github user zero323 commented on the issue: https://github.com/apache/spark/pull/17825 I wonder if it would make more sense to make `alias` generic for both `object` and `data`: signature(object = "SparkDataFrame", data = "character") and skip the type checks.

[GitHub] spark pull request #17852: [SPARK-20546][Deploy] spark-class gets syntax err...

2017-05-03 Thread jyu00
GitHub user jyu00 opened a pull request: https://github.com/apache/spark/pull/17852 [SPARK-20546][Deploy] spark-class gets syntax error in posix mode ## What changes were proposed in this pull request? Updated spark-class to turn off posix mode so the process substitution

[GitHub] spark issue #17852: [SPARK-20546][Deploy] spark-class gets syntax error in p...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17852 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17847: [SPARK-20590] Map default input data source forma...

2017-05-03 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/17847#discussion_r114683203 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -523,7 +530,8 @@ object DataSource {

[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register

2017-05-03 Thread zero323
Github user zero323 commented on the issue: https://github.com/apache/spark/pull/17831 I think there is no conflict between #17848 and this. As of 2.2 we no longer return `UserDefinedFunction` from `udf` (and we never documented `UserDefinedFunctions`) so changes will have to be

[GitHub] spark pull request #17847: [SPARK-20590] Map default input data source forma...

2017-05-03 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/17847#discussion_r114683317 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -483,35 +483,42 @@ case class DataSource(

[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...

2017-05-03 Thread mgummelt
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 > a) we have explicitly based our support on it What does this mean? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/17540 I'm not an expert on the metrics path, but I think we should be able to join up the actual physical plans well enough to display everything. I doubt it will be a long-term regression, but I don't

[GitHub] spark issue #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17851 **[Test build #76436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76436/testReport)** for PR 17851 at commit

[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...

2017-05-03 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 > This was my point - we should not introduce system specific api's into spark core infrastructure api's/spi's Sorry, I still have no idea what your point is. How do you suggest we support

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17540 I was not saying there is no way to fix metrics. Just asking your thoughts. If we don't have a concrete plan, it might be a long-term regression if just merging this PR. I just want to

[GitHub] spark issue #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17851 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17851 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76435/ Test FAILed. ---

[GitHub] spark issue #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17851 **[Test build #76435 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76435/testReport)** for PR 17851 at commit

[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...

2017-05-03 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17723 @vanzip wrote: > So, this is purely about handling Hadoop authentication for Hadoop services. This was my point - we should not introduce system specific api's into spark core

[GitHub] spark issue #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17851 **[Test build #76435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76435/testReport)** for PR 17851 at commit

[GitHub] spark pull request #17851: [SPARK-20585][SPARKR] R generic hint support

2017-05-03 Thread zero323
GitHub user zero323 opened a pull request: https://github.com/apache/spark/pull/17851 [SPARK-20585][SPARKR] R generic hint support ## What changes were proposed in this pull request? Adds support for generic hints on `SparkDataFrame` ## How was this patch tested?

[GitHub] spark issue #17847: [SPARK-20590] Map default input data source formats to i...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17847 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76427/ Test PASSed. ---

[GitHub] spark issue #17847: [SPARK-20590] Map default input data source formats to i...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17847 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17847: [SPARK-20590] Map default input data source formats to i...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17847 **[Test build #76427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76427/testReport)** for PR 17847 at commit

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/17540 @zsxwing, you don't think there's a way to fix metrics? I don't know exactly how to fix the UI to show two plans worth of metrics, but it seems like it can be done. What about also updating

[GitHub] spark issue #17793: [SPARK-20484][MLLIB] Add documentation to ALS code

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17793 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76430/ Test PASSed. ---

[GitHub] spark issue #17793: [SPARK-20484][MLLIB] Add documentation to ALS code

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17793 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17793: [SPARK-20484][MLLIB] Add documentation to ALS code

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17793 **[Test build #76430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76430/testReport)** for PR 17793 at commit

[GitHub] spark issue #17848: [SPARK-20586] [SQL] Add deterministic and distinctLike t...

2017-05-03 Thread zero323
Github user zero323 commented on the issue: https://github.com/apache/spark/pull/17848 Disabling optimizations aside, to what extent can we actually support nondeterministic functions? Right now a common user mistake is to run RNG inside an UDF. `nonDeterministic`could suggest it is

[GitHub] spark issue #17850: [SPARK-20584][PYSPARK][SQL] Python generic hint support

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17850 **[Test build #76433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76433/testReport)** for PR 17850 at commit

[GitHub] spark issue #17850: [SPARK-20584][PYSPARK][SQL] Python generic hint support

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17850 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76433/ Test PASSed. ---

[GitHub] spark issue #17850: [SPARK-20584][PYSPARK][SQL] Python generic hint support

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17850 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17540 > @zsxwing, I don't know. Sounds like we should fix the underlying problem that there are 2 physical plans. SQL metrics won't work without fixing it. IMO, that's more serious than the

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/17540 @zsxwing, I don't know. Sounds like we should fix the underlying problem that there are 2 physical plans. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register

2017-05-03 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17831 Yea, thanks for chiming in. It helped me a lot to understand the context. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17540 > That requires breaking the command into two phases, one to get a SparkPlan and one to run it. Yeah, but how to show metrics you get from a plan on another plan's DAG considering these

[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register

2017-05-03 Thread zero323
Github user zero323 commented on the issue: https://github.com/apache/spark/pull/17831 Thanks @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/17540 @zsxwing, there should be a fix for the metrics without waiting for all of the bad plans to be fixed (which is to basically eliminate the use of `ExecutedCommandExec`). The metrics are

[GitHub] spark issue #17346: [SPARK-19965][SS] DataFrame batch reader may fail to inf...

2017-05-03 Thread lw-lin
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/17346 thank you @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #17540: [SPARK-20213][SQL][UI] Fix DataFrameWriter operations in...

2017-05-03 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/17540 @rdblue I just tested this PR and found that I could not see any SQL metrics on Web UI. This is pretty important for many users to analyze their queries. What's your plan to fix it? As far

[GitHub] spark issue #17834: [SPARK-7481] [build] Add spark-hadoop-cloud module to pu...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17834 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17834: [SPARK-7481] [build] Add spark-hadoop-cloud module to pu...

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17834 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76426/ Test PASSed. ---

[GitHub] spark issue #17834: [SPARK-7481] [build] Add spark-hadoop-cloud module to pu...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17834 **[Test build #76426 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76426/testReport)** for PR 17834 at commit

[GitHub] spark issue #17100: [SPARK-13947][SQL] PySpark DataFrames: The error message...

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17100 **[Test build #76434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76434/testReport)** for PR 17100 at commit

[GitHub] spark issue #17850: [SPARK-20584][PYSPARK][SQL] Python generic hint support

2017-05-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17850 **[Test build #76433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76433/testReport)** for PR 17850 at commit

[GitHub] spark issue #17850: [SPARK-20584][PYSPARK][SQL] Python generic hint support

2017-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17850 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

  1   2   3   4   >