[GitHub] spark issue #16486: [SPARK-13610][ML] Create a Transformer to disassemble ve...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16486 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #16484: [SPARK-19083]sbin/start-history-server.sh script ...

2017-01-05 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/16484#discussion_r94910539 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala --- @@ -445,12 +445,24 @@ private[deploy] class Worker( // Create

[GitHub] spark pull request #16486: [SPARK-13610][ML] Create a Transformer to disasse...

2017-01-05 Thread leonfl
GitHub user leonfl opened a pull request: https://github.com/apache/spark/pull/16486 [SPARK-13610][ML] Create a Transformer to disassemble vectors in Data… JIRA Issue: https://issues.apache.org/jira/browse/SPARK-13610 ## What changes were proposed in this pull request?

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14451 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70966/ Test FAILed. ---

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14451 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14451 **[Test build #70966 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70966/testReport)** for PR 14451 at commit

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14451 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70965/ Test FAILed. ---

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14451 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14451 **[Test build #70965 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70965/testReport)** for PR 14451 at commit

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94909141 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/ProjectEstimation.scala --- @@ -0,0 +1,44 @@ +/* + *

[GitHub] spark issue #16485: [SPARK-19099] correct the wrong time display in history ...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16485 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #16485: [SPARK-19099] correct the wrong time display in h...

2017-01-05 Thread 351zyf
GitHub user 351zyf opened a pull request: https://github.com/apache/spark/pull/16485 [SPARK-19099] correct the wrong time display in history server web UI JIRA Issue: https://issues.apache.org/jira/browse/SPARK-19099 Correct the wrong job start/end time display in spark

[GitHub] spark issue #16484: Sh

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16484 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16480: [SPARK-18194][ML] Log instrumentation in OneVsRest, Cros...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16480 **[Test build #70972 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70972/testReport)** for PR 16480 at commit

[GitHub] spark pull request #16484: Sh

2017-01-05 Thread zuotingbing
GitHub user zuotingbing opened a pull request: https://github.com/apache/spark/pull/16484 Sh JIRA Issue: https://issues.apache.org/jira/browse/SPARK-19083# sbin/start-history-server.sh script use of $@ without quotes, this will affect the length of args which used in

[GitHub] spark issue #16483: [SPARK-18847][GraphX] PageRank gives incorrect results f...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16483 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70969/ Test PASSed. ---

[GitHub] spark issue #16483: [SPARK-18847][GraphX] PageRank gives incorrect results f...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16483 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16483: [SPARK-18847][GraphX] PageRank gives incorrect results f...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16483 **[Test build #70969 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70969/testReport)** for PR 16483 at commit

[GitHub] spark issue #16479: [SPARK-19085][SQL] cleanup OutputWriterFactory and Outpu...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16479 **[Test build #70971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70971/testReport)** for PR 16479 at commit

[GitHub] spark issue #16480: [SPARK-18194][ML] Log instrumentation in OneVsRest, Cros...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16480 **[Test build #70970 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70970/testReport)** for PR 16480 at commit

[GitHub] spark issue #16480: [SPARK-18194][ML] Log instrumentation in OneVsRest, Cros...

2017-01-05 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16480 LGTM pending Jenkins tests Thanks @sueann ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16480: [SPARK-18194][ML] Log instrumentation in OneVsRest, Cros...

2017-01-05 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16480 add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #16480: [SPARK-18194][ML] Log instrumentation in OneVsRest, Cros...

2017-01-05 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16480 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark pull request #16479: [SPARK-19085][SQL] cleanup OutputWriterFactory an...

2017-01-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16479#discussion_r94907874 --- Diff: mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala --- @@ -45,9 +45,12 @@ private[libsvm] class LibSVMOutputWriter(

[GitHub] spark pull request #16240: [SPARK-16792][SQL] Dataset containing a Case Clas...

2017-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16240 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #16240: [SPARK-16792][SQL] Dataset containing a Case Class with ...

2017-01-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16240 @michalsenkyr please create 2 more tickets for the optimization you metioned in https://github.com/apache/spark/pull/16240#issuecomment-266318016 and the nested custom collection problem. ---

[GitHub] spark issue #16240: [SPARK-16792][SQL] Dataset containing a Case Class with ...

2017-01-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16240 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16430: [SPARK-17077] [SQL] Cardinality estimation for project o...

2017-01-05 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16430 Thanks for review! I'll fix it today --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory ...

2017-01-05 Thread merlintang
Github user merlintang commented on a diff in the pull request: https://github.com/apache/spark/pull/15819#discussion_r94906952 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala --- @@ -216,5 +219,37 @@ class VersionsSuite extends SparkFunSuite

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread wzhfy
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94906707 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/ProjectEstimation.scala --- @@ -0,0 +1,44 @@ +/* + *

[GitHub] spark issue #16483: [SPARK-18847][GraphX] PageRank gives incorrect results f...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16483 **[Test build #70969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70969/testReport)** for PR 16483 at commit

[GitHub] spark pull request #16431: [SPARK-19020] [SQL] Cardinality estimation of agg...

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16431#discussion_r94906692 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/AggregateEstimation.scala --- @@ -0,0 +1,59 @@ +/* + *

[GitHub] spark issue #16429: [SPARK-19019][PYTHON] Fix hijacked `collections.namedtup...

2017-01-05 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/16429 Yah I think the third party library is definitly a follow up type task, but something we should consider after this since we are back porting fixes from another repo into our own code :) --- If

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94906528 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/FilterEstimation.scala --- @@ -0,0 +1,479 @@ +/* + *

[GitHub] spark pull request #16483: [SPARK-18847][GraphX] PageRank gives incorrect re...

2017-01-05 Thread aray
GitHub user aray opened a pull request: https://github.com/apache/spark/pull/16483 [SPARK-18847][GraphX] PageRank gives incorrect results for graphs with sinks ## What changes were proposed in this pull request? Graphs with sinks (vertices with no outgoing edges) don't have

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94906278 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/FilterEstimation.scala --- @@ -0,0 +1,479 @@ +/* + *

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94906270 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/FilterEstimation.scala --- @@ -0,0 +1,479 @@ +/* + *

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94906266 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/FilterEstimation.scala --- @@ -0,0 +1,479 @@ +/* + *

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94906221 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/FilterEstimation.scala --- @@ -0,0 +1,479 @@ +/* + *

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94906180 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/Range.scala --- @@ -0,0 +1,75 @@ +/* + * Licensed to the

[GitHub] spark issue #16429: [SPARK-19019][PYTHON] Fix hijacked `collections.namedtup...

2017-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16429 BTW, this at least makes Spark on working state in Python 3.6.0 with a minimised changes. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94906142 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/Range.scala --- @@ -0,0 +1,75 @@ +/* + * Licensed to the

[GitHub] spark issue #16429: [SPARK-19019][PYTHON] Fix hijacked `collections.namedtup...

2017-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16429 @holdenk, Yup. (actually, it is cloudpipe/cloudpickle@cbd3f34 because there were more minor fixes). It seems there are several customized codes such as injecting `numpy` as below:

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94905939 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/FilterEstimation.scala --- @@ -0,0 +1,479 @@ +/* + *

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94905864 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/FilterEstimation.scala --- @@ -0,0 +1,479 @@ +/* + *

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94905897 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/FilterEstimation.scala --- @@ -0,0 +1,479 @@ +/* + *

[GitHub] spark pull request #16395: [SPARK-17075][SQL] implemented filter estimation

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16395#discussion_r94905771 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/estimation/FilterEstimationSuite.scala --- @@ -0,0 +1,226 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14451 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14451 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70961/ Test PASSed. ---

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14451 **[Test build #70961 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70961/testReport)** for PR 14451 at commit

[GitHub] spark issue #16429: [SPARK-19019][PYTHON] Fix hijacked `collections.namedtup...

2017-01-05 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/16429 (We might want to consider using cloudpickle as a library since others are working on maintaining it and they had a fix for 3.6 support back in early November, but that would involve adding another

[GitHub] spark issue #16429: [SPARK-19019][PYTHON] Fix hijacked `collections.namedtup...

2017-01-05 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/16429 Am I correct in assuming this is based on https://github.com/cloudpipe/cloudpickle/commit/4945361c2db92095f934b92a6c00316243caf3cc ? --- If your project is set up for it, you can reply to this

[GitHub] spark pull request #16464: [SPARK-19066][SparkR]:SparkR LDA doesn't set opti...

2017-01-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16464#discussion_r94905455 --- Diff: mllib/src/main/scala/org/apache/spark/ml/r/LDAWrapper.scala --- @@ -172,6 +187,8 @@ private[r] object LDAWrapper extends

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94905442 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/estimation/EstimationSuite.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...

2017-01-05 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16401 How about the following? ``` def stats(conf: CatalystConf): Statistics = statsCache.getOrElse { invalidateStatsCache() statsCache = Some(computeStats(conf)) statsCache.get

[GitHub] spark pull request #16464: [SPARK-19066][SparkR]:SparkR LDA doesn't set opti...

2017-01-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16464#discussion_r94905327 --- Diff: R/pkg/inst/tests/testthat/test_mllib.R --- @@ -907,6 +917,8 @@ test_that("spark.lda with text input", { expect_equal(logPerplexity,

[GitHub] spark pull request #16464: [SPARK-19066][SparkR]:SparkR LDA doesn't set opti...

2017-01-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16464#discussion_r94905196 --- Diff: mllib/src/main/scala/org/apache/spark/ml/r/LDAWrapper.scala --- @@ -123,6 +126,10 @@ private[r] object LDAWrapper extends

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #70968 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70968/testReport)** for PR 13599 at commit

[GitHub] spark pull request #16454: [SPARK-19055][SQL][PySpark] Fix SparkSession init...

2017-01-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16454#discussion_r94905140 --- Diff: python/pyspark/sql/session.py --- @@ -214,8 +214,14 @@ def __init__(self, sparkContext, jsparkSession=None): self._wrapped =

[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...

2017-01-05 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/16401 do you mean no cache for `def statistics(conf)`: ``` def statistics(conf: CatalystConf): Statistics = { if(conf.cboEnabled) newStats else oldStats } lazy val newStats... lazy

[GitHub] spark issue #16482: [SPARK-19038][YARN] Fix for misnamed keytab in app stagi...

2017-01-05 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/16482 I think there's no issue in yarn side, please check the yarn container's local cache: ``` -rw--- 1 spark hadoop 200 Jan 6 05:59 container_tokens -rwx-- 1 spark hadoop

[GitHub] spark issue #12257: [SPARK-14483][WEBUI] Display user name for each job and ...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12257 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #16454: [SPARK-19055][SQL][PySpark] Fix SparkSession init...

2017-01-05 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/16454#discussion_r94904472 --- Diff: python/pyspark/sql/session.py --- @@ -214,8 +214,14 @@ def __init__(self, sparkContext, jsparkSession=None): self._wrapped =

[GitHub] spark issue #12257: [SPARK-14483][WEBUI] Display user name for each job and ...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12257 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70958/ Test PASSed. ---

[GitHub] spark issue #12257: [SPARK-14483][WEBUI] Display user name for each job and ...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12257 **[Test build #70958 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70958/testReport)** for PR 12257 at commit

[GitHub] spark issue #16441: [SPARK-14975][ML][WIP] Fixed GBTClassifier to predict pr...

2017-01-05 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/16441 It looks like I am failing the binary compatibility tests despite this constructor being private: class GBTClassificationModel private[ml]( @Since("1.6.0") override val uid:

[GitHub] spark issue #16478: [WIP][SQL] Revise user defined types (UDT)

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16478 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70951/ Test FAILed. ---

[GitHub] spark issue #16478: [WIP][SQL] Revise user defined types (UDT)

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16478 **[Test build #70951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70951/testReport)** for PR 16478 at commit

[GitHub] spark issue #16478: [WIP][SQL] Revise user defined types (UDT)

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16478 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14451 **[Test build #70967 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70967/testReport)** for PR 14451 at commit

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14451 **[Test build #70966 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70966/testReport)** for PR 14451 at commit

[GitHub] spark pull request #16454: [SPARK-19055][SQL][PySpark] Fix SparkSession init...

2017-01-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16454#discussion_r94903491 --- Diff: python/pyspark/sql/session.py --- @@ -214,8 +214,14 @@ def __init__(self, sparkContext, jsparkSession=None): self._wrapped =

[GitHub] spark pull request #14451: [SPARK-16848][SQL] Check schema validation for us...

2017-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/14451#discussion_r94903467 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -536,10 +539,17 @@ class DataFrameReader private[sql](sparkSession:

[GitHub] spark issue #14451: [SPARK-16848][SQL] Check schema validation for user-spec...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14451 **[Test build #70965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70965/testReport)** for PR 14451 at commit

[GitHub] spark pull request #14451: [SPARK-16848][SQL] Check schema validation for us...

2017-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/14451#discussion_r94903385 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -536,10 +539,17 @@ class DataFrameReader private[sql](sparkSession:

[GitHub] spark pull request #16454: [SPARK-19055][SQL][PySpark] Fix SparkSession init...

2017-01-05 Thread vijoshi
Github user vijoshi commented on a diff in the pull request: https://github.com/apache/spark/pull/16454#discussion_r94903274 --- Diff: python/pyspark/sql/session.py --- @@ -214,8 +214,14 @@ def __init__(self, sparkContext, jsparkSession=None): self._wrapped =

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94903216 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/estimation/EstimationSuite.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #14451: [SPARK-16848][SQL] Check schema validation for us...

2017-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/14451#discussion_r94903231 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -536,10 +539,17 @@ class DataFrameReader private[sql](sparkSession:

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94903188 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/estimation/EstimationSuite.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16401: [SPARK-18998] [SQL] Add a cbo conf to switch between def...

2017-01-05 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16401 @wzhfy I thought about this more. Why don't we just get rid of the existing "def statistics", and keep only one function ``` def statistics(conf: CatalystConf): Statistics ```

[GitHub] spark issue #9920: [SPARK-11569] [ML] Fix StringIndexer to handle null value...

2017-01-05 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/9920 @jliwork @srowen are you currently working on this in-progress JIRA 11569? If not, I would be interested in continuing the initial pull request that was closed. Please let me know, thank

[GitHub] spark pull request #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory ...

2017-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15819#discussion_r94902388 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala --- @@ -216,5 +219,37 @@ class VersionsSuite extends SparkFunSuite

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94902185 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/ProjectEstimation.scala --- @@ -0,0 +1,44 @@ +/* + *

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70964 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70964/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15819 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70964/ Test FAILed. ---

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94902161 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/ProjectEstimation.scala --- @@ -0,0 +1,44 @@ +/* + *

[GitHub] spark issue #14451: [SPARK-16848][SQL] Make jdbc() and read.format("jdbc") c...

2017-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14451 Yeap. Then, let me add a test case and fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94902084 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/EstimationUtils.scala --- @@ -0,0 +1,54 @@ +/* + *

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94902048 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/estimation/EstimationUtils.scala --- @@ -0,0 +1,54 @@ +/* + *

[GitHub] spark pull request #16441: [SPARK-14975][ML][WIP] Fixed GBTClassifier to pre...

2017-01-05 Thread sethah
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/16441#discussion_r94902030 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala --- @@ -248,12 +269,38 @@ class GBTClassificationModel private[ml](

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15819 **[Test build #70964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70964/consoleFull)** for PR 15819 at commit

[GitHub] spark issue #15819: [SPARK-18372][SQL][Branch-1.6].Staging directory fail to...

2017-01-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15819 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #16441: [SPARK-14975][ML][WIP] Fixed GBTClassifier to pre...

2017-01-05 Thread sethah
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/16441#discussion_r94901794 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala --- @@ -248,12 +269,38 @@ class GBTClassificationModel private[ml](

[GitHub] spark pull request #16430: [SPARK-17077] [SQL] Cardinality estimation for pr...

2017-01-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/16430#discussion_r94901694 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/estimation/EstimationSuite.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16482: [SPARK-19038][YARN] Fix for misnamed keytab in app stagi...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16482 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #16482: [SPARK-19038][YARN] Fix for misnamed keytab in ap...

2017-01-05 Thread parente
GitHub user parente opened a pull request: https://github.com/apache/spark/pull/16482 [SPARK-19038][YARN] Fix for misnamed keytab in app staging dir ## What changes were proposed in this pull request? Bug fix to respect the generate AM keytab name when copying the local

[GitHub] spark issue #14451: [SPARK-16848][SQL] Make jdbc() and read.format("jdbc") c...

2017-01-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14451 That assumes `JDBCRelation` does not extend `SchemaRelationProvider`. Yeah. That is right so far, but we do not know whether this might be changed. Normally, the best way is to add a test case

[GitHub] spark issue #14451: [SPARK-16848][SQL] Make jdbc() and read.format("jdbc") c...

2017-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14451 Sure, I will add. It seems `format("jdbc").load()` thorws an exception already which is being tested

[GitHub] spark issue #16481: [SPARK-19092] [SQL] Save() API of DataFrameWriter should...

2017-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16481 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70956/ Test PASSed. ---

  1   2   3   4   5   6   >