[GitHub] spark pull request: [ML] [R] SparkR::glm summary result to compare...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9590#issuecomment-155349937 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11624][SQL]fix commands that need hive ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9589#issuecomment-155349933 **[Test build #45508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45508/consoleFull)** for PR 9589 at commit [`e1716e2`](https://github.com/apache/spark/commit/e1716e22a6fd50f35cc2c5e8362ee0828c9622c9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [ML] [R] SparkR::glm summary result to compare...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9590#issuecomment-155349949 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [ML] [R] SparkR::glm summary result to compare...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/9590#discussion_r44378156
--- Diff: R/pkg/R/mllib.R ---
@@ -106,7 +106,7 @@ setMethod("summary", signature(object =
"PipelineModel"),
coefficients <- matrix(coefficients, ncol = 4)
colnames(coefficients) <- c("Estimate", "Std. Error", "t
value", "Pr(>|t|)")
rownames(coefficients) <- unlist(features)
- return(list(DevianceResiduals = devianceResiduals,
Coefficients = coefficients))
+ return(list(devianceResiduals = devianceResiduals,
coefficients = coefficients))
--- End diff --
Rename these variables with lowercase ones which are consistent with R.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11206] Support SQL UI on the history se...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9297#issuecomment-155349864 **[Test build #45507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45507/consoleFull)** for PR 9297 at commit [`51f913b`](https://github.com/apache/spark/commit/51f913bed91dd95029b404092e8cd7c25b02cad6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL][follow-up] clean up java tu...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9567#issuecomment-155349603 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL][follow-up] clean up java tu...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9567#issuecomment-155349606 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45493/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [ML] [R] SparkR::glm summary result to compare...
GitHub user yanboliang opened a pull request: https://github.com/apache/spark/pull/9590 [ML] [R] SparkR::glm summary result to compare with native R Due to [SPARK-11587](https://issues.apache.org/jira/browse/SPARK-11587) has been fixed, we should compare SparkR::glm summary result with native R output rather than hard-code one. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yanboliang/spark glm-r-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9590.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9590 commit 2975ab12228dd6975ace1c7b33544c21e6a65452 Author: Yanbo Liang Date: 2015-11-10T07:50:10Z change glm summary result to compare with native R --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11564][SQL][follow-up] clean up java tu...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9567#issuecomment-155349505 **[Test build #45493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45493/consoleFull)** for PR 9567 at commit [`fb1fe8f`](https://github.com/apache/spark/commit/fb1fe8f2b87df9e84ab794c24dbf965affb28cb4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11624][SQL]fix commands that need hive ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9589#issuecomment-155349061 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11624][SQL]fix commands that need hive ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9589#issuecomment-155349036 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11396] [SQL] add native implementation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9347#issuecomment-155349044 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11396] [SQL] add native implementation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9347#issuecomment-155349059 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11624][SQL]fix commands that need hive ...
GitHub user adrian-wang opened a pull request: https://github.com/apache/spark/pull/9589 [SPARK-11624][SQL]fix commands that need hive to exec spark-sql> !echo "test"; Exception in thread "main" java.lang.ClassCastException: org.apache.hadoop.hive.ql.session.SessionState cannot be cast to org.apache.hadoop.hive.cli.CliSessionState at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:112) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:301) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:242) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:691) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) You can merge this pull request into a Git repository by running: $ git pull https://github.com/adrian-wang/spark clicommand Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9589.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9589 commit e1716e22a6fd50f35cc2c5e8362ee0828c9622c9 Author: Daoyuan Wang Date: 2015-11-10T07:49:24Z fix commands that need hive to exec --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9830] [SQL] Remove AggregateExpression1...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9556#issuecomment-155348763 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9830] [SQL] Remove AggregateExpression1...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9556#issuecomment-155348765 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45492/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9830] [SQL] Remove AggregateExpression1...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9556#issuecomment-155348607 **[Test build #45492 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45492/consoleFull)** for PR 9556 at commit [`16826e6`](https://github.com/apache/spark/commit/16826e6d5cb9636b6a72125fcccfc4273227c45b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11206] Support SQL UI on the history se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9297#issuecomment-155348295 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11206] Support SQL UI on the history se...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9297#issuecomment-155348318 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11206] Support SQL UI on the history se...
Github user carsonwang commented on the pull request: https://github.com/apache/spark/pull/9297#issuecomment-155348070 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818] Re-enable Docker tests for JDBC d...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9503#issuecomment-155347761 Created https://tachyon.atlassian.net/browse/TACHYON-1254 for this; this will be worked around in Spark via my `lib_managed` patch, but I still feel that it should be fixed in Tachyon as well so that it doesn't bite anyone in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() (New v...
Github user olarayej commented on the pull request: https://github.com/apache/spark/pull/9579#issuecomment-155347716 @felixcheung @sun-rui All set. Can we please merge? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818] Re-enable Docker tests for JDBC d...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9503#issuecomment-155346029 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45495/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818] Re-enable Docker tests for JDBC d...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9503#issuecomment-155346028 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818] Re-enable Docker tests for JDBC d...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9503#issuecomment-155345975 **[Test build #45495 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45495/consoleFull)** for PR 9503 at commit [`9011bc5`](https://github.com/apache/spark/commit/9011bc5addbf97543e93790da2027685ad6f661e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11259] [ML] Params.validateParams() sho...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9224#issuecomment-155345652 **[Test build #45506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45506/consoleFull)** for PR 9224 at commit [`819f9ca`](https://github.com/apache/spark/commit/819f9caaa6d5d02c925f103b2f7b281381391655). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818] Re-enable Docker tests for JDBC d...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9503#issuecomment-155344851 I found this Manifest class-path issue to be extremely puzzling, but it looks like other people have hit it as well: https://dzone.com/articles/jar-manifest-class-path-is-not-for-java-applicatio Quote: > A JAR with a Class-Path manifest header will make those classpath entries available to the Java classloader regardless of whether that JAR is executed with java -jar ... or is simply placed on the classpath of a larger Java application. This puzzling behavior is described a bit more explicitly in [Java 8's JAR specification](https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#classpath). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11598] [SQL] enable tests for ShuffledH...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9573 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11599] [SQL] fix NPE when resolve Hive ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9576 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11463] [PYSPARK] only install signal in...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/9574#issuecomment-155344192 cc @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11599] [SQL] fix NPE when resolve Hive ...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/9576#issuecomment-155344051 Merged into master and 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155343561 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45505/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11259] [ML] Params.validateParams() sho...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9224#issuecomment-155343642 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155343556 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11259] [ML] Params.validateParams() sho...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9224#issuecomment-155343549 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155343550 **[Test build #45505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45505/consoleFull)** for PR 7888 at commit [`27faa6b`](https://github.com/apache/spark/commit/27faa6b70e8332f6f70359739cb417be5be7e31e). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11259] [ML] Params.validateParams() sho...
Github user yanboliang commented on the pull request: https://github.com/apache/spark/pull/9224#issuecomment-155342951 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() (New v...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9579#issuecomment-155342496 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45499/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() (New v...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9579#issuecomment-155342487 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155342077 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() (New v...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9579#issuecomment-155342110 **[Test build #45499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45499/consoleFull)** for PR 9579 at commit [`01366a6`](https://github.com/apache/spark/commit/01366a6a83ca26da6d6e1233306cf6148d5d051f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155342083 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45504/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818] Re-enable Docker tests for JDBC d...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/9503#issuecomment-155341829 I only reviewed the build changes, but they look good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155341339 **[Test build #45505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45505/consoleFull)** for PR 7888 at commit [`27faa6b`](https://github.com/apache/spark/commit/27faa6b70e8332f6f70359739cb417be5be7e31e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7841][BUILD] Stop using retrieveManaged...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/9575#issuecomment-155341204 It's hard for me to rule out that there is _no_ other reason lib_managed is used at present. I audited all the uses of it I could find in the codebase and it appears they all relate to the DataNucleus jars. So LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155335196 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155335140 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155334978 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155334981 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45502/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155334934 **[Test build #45502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45502/consoleFull)** for PR 9287 at commit [`fe9de83`](https://github.com/apache/spark/commit/fe9de83ec881d8c021a4f2b790dcb05efa6a8ce0). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `trait PasswordAuthenticator `\n * `class ExternalShuffleService(`\n * `public class ClientAddress implements Serializable `\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818] Re-enable Docker tests for JDBC d...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/9503#issuecomment-155334719 In fact, `httpcore-4.2.4` doesn't even show up in the `-classpath` argument passed to `ForkMain`. I used some of the handy classpath-debugging tools in [JHades](https://github.com/jhades/jhades) to dump more information and it looks like `httpcore-4.2.4` is included in the Tachyon JARs' `MANIFEST.MF` files. Here's the manfest from `tachyon-client`: ``` Manifest-Version: 1.0 Built-By: DreamSweet Build-Jdk: 1.7.0_75 Class-Path: guava-14.0.1.jar commons-lang-2.4.jar commons-io-2.4.jar c ommons-lang3-3.0.jar curator-framework-2.1.0-incubating.jar zookeeper -3.4.5.jar slf4j-log4j12-1.7.2.jar log4j-1.2.15.jar mail-1.4.jar acti vation-1.1.jar jline-0.9.94.jar netty-3.2.2.Final.jar curator-client- 2.1.0-incubating.jar curator-recipes-2.1.0-incubating.jar slf4j-api-1 .7.2.jar hadoop-client-1.0.4.jar hadoop-core-1.0.4.jar xmlenc-0.52.ja r commons-codec-1.4.jar commons-math-2.1.jar commons-configuration-1. 6.jar commons-collections-3.2.1.jar commons-digester-1.8.jar commons- beanutils-1.7.0.jar commons-beanutils-core-1.8.0.jar commons-net-1.4. 1.jar commons-el-1.0.jar hsqldb-1.8.0.10.jar oro-2.0.8.jar jackson-ma pper-asl-1.0.1.jar jackson-core-asl-1.0.1.jar tachyon-common-0.8.1.ja r netty-all-4.0.28.Final.jar libthrift-0.9.2.jar httpclient-4.2.5.jar httpcore-4.2.4.jar tachyon-client-unshaded-0.8.1.jar tachyon-underfs -hdfs-0.8.1.jar tachyon-underfs-glusterfs-0.8.1.jar glusterfs-hadoop- 2.3.13.jar hadoop-yarn-server-nodemanager-2.3.0.jar hadoop-yarn-commo n-2.3.0.jar hadoop-yarn-api-2.3.0.jar jaxb-api-2.2.2.jar stax-api-1.0 -2.jar jettison-1.1.jar servlet-api-2.5.jar jersey-core-1.9.jar jerse y-client-1.9.jar jetty-util-6.1.26.jar hadoop-annotations-2.3.0.jar g uice-servlet-3.0.jar protobuf-java-2.5.0.jar guice-3.0.jar javax.inje ct-1.jar aopalliance-1.0.jar jersey-json-1.9.jar jaxb-impl-2.2.3-1.ja r jackson-jaxrs-1.8.3.jar jackson-xc-1.8.3.jar jersey-guice-1.9.jar h adoop-yarn-server-common-2.3.0.jar hadoop-common-2.3.0.jar commons-cl i-1.2.jar commons-math3-3.1.1.jar jetty-6.1.26.jar jersey-server-1.9. jar asm-3.1.jar jasper-compiler-5.5.23.jar jasper-runtime-5.5.23.jar jsp-api-2.1.jar avro-1.7.4.jar paranamer-2.3.jar snappy-java-1.0.4.1. jar hadoop-auth-2.3.0.jar jsch-0.1.42.jar jsr305-1.3.9.jar commons-co mpress-1.4.1.jar xz-1.0.jar tachyon-underfs-s3-0.8.1.jar jets3t-0.8.1 .jar commons-logging-1.1.1.jar commons-httpclient-3.1.jar java-xmlbui lder-0.4.jar tachyon-underfs-local-0.8.1.jar Created-By: Apache Maven 3.3.3 Archiver-Version: Plexus Archiver ``` According to the [Java 7 JAR specification](https://docs.oracle.com/javase/7/docs/technotes/guides/jar/jar.html#Manifest_Specification) (emphasis mine): > Class-Path: The value of this attribute specifies the **relative** URLs of the extensions or libraries that this application or extension needs. URLs are separated by one or more spaces. The application or extension class loader uses the value of this attribute to construct its internal search path. Assuming that "relative" means "relative to the JAR containing the manifest", I think that what's happening is that this is causing the `httpcore-4.2.4` JAR from `lib_managed` to be added to the classpath, which explains why testing this patch after rebasing on top of #9575 (which minimizes our use of `lib_managed/jars` to only include the Datanucleus JARs) seemed to fix things. What a giant mess. It looks like the `` manifest configuration ended up being added to Tachyon's root POM as part of https://github.com/amplab/tachyon/pull/939. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6152] Use shaded ASM5 to support closur...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/9512#issuecomment-155334662 This looks fine (i.e. LGTM). However, we could also look into actually shading asm ourselves in our published artifacts, similar to how we now shade jetty and other things. I'm fine to just use this already shaded one too though. Seems to do no harm and will allow us to work with Java 8. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11360] [Doc] Loss of nullability when w...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/9314#issuecomment-155334571 Got it, thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155333810 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155333879 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9165#issuecomment-155334057 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45501/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9165#issuecomment-155334055 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11593][SQL] Replace catalyst converter ...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9565#discussion_r44375480
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -60,924 +77,238 @@ case class ScalaUDF(
*/
- // Accessors used in genCode
- def userDefinedFunc(): AnyRef = function
- def getChildren(): Seq[Expression] = children
-
- private[this] val f = children.size match {
-case 0 =>
- val func = function.asInstanceOf[() => Any]
- (input: InternalRow) => {
-func()
- }
-
-case 1 =>
- val func = function.asInstanceOf[(Any) => Any]
- val child0 = children(0)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)))
- }
-
-case 2 =>
- val func = function.asInstanceOf[(Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)))
- }
-
-case 3 =>
- val func = function.asInstanceOf[(Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)))
- }
-
-case 4 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)),
- converter3(child3.eval(input)))
- }
-
-case 5 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- val child4 = children(4)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- lazy val converter4 =
CatalystTypeConverters.createToScalaConverter(child4.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)),
- converter3(child3.eval(input)),
- converter4(child4.eval(input)))
- }
-
-case 6 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any, Any, Any) =>
Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- val child4 = children(4)
- val child5 = children(5)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- lazy val converter4 =
CatalystTypeConverters.createToScalaConverter(child4.dataType)
- lazy val converter5 =
Ca
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user GraceH commented on the pull request: https://github.com/apache/spark/pull/7888#issuecomment-155333294 @vanzin Got your point. I will follow that by eliminating the secondary option in public API. thanks for the confirmation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9165#issuecomment-155333268 **[Test build #45503 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45503/consoleFull)** for PR 9165 at commit [`290f798`](https://github.com/apache/spark/commit/290f7989e2646d242014f40b3937aefbf3e280bc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9552] Add force control for killExecuto...
Github user GraceH commented on a diff in the pull request:
https://github.com/apache/spark/pull/7888#discussion_r44375390
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -509,6 +511,13 @@ private[spark] class ExecutorAllocationManager(
private def onExecutorBusy(executorId: String): Unit = synchronized {
logDebug(s"Clearing idle timer for $executorId because it is now
running a task")
removeTimes.remove(executorId)
+
+// Executor is added to remove by misjudgment due to async listener
making it as idle).
+// see SPARK-9552
+if (executorsPendingToRemove.contains(executorId)) {
--- End diff --
@vanzin, In the original design, I changed the return back value for that
function (`killExecutors`). Not only for it is the last round of review
comments. But also since It is still a little bit strange. For example, you
have 3 executors to kill with force=false. And you find one of them is busy. It
is hard to tell killing success or not directly. But if we only support single
executor here, it is much simple and straightforward.
Besides, this is changed according to last round of review comments. Since
the killExecutors only returns with the acknowledge (in documentation), which
doesn't indicate the status of kill action. Please let me know your further
thoughts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11396] [SQL] add native implementation ...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/9347#issuecomment-155331482 LGTM overall, just one minor comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11396] [SQL] add native implementation ...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9347#discussion_r44375337
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
---
@@ -299,7 +299,20 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
}
/**
- * Converts time string with given pattern
+ * Converts time string with given pattern.
+ * Deterministic version of [[UnixTimestamp]], must have at least one
parameter.
+ */
+case class ToUnixTimestamp(timeExp: Expression, format: Expression)
extends UnixTime {
+ def this(time: Expression) = {
+this(time, Literal("-MM-dd HH:mm:ss"))
+ }
+
+ override def left: Expression = timeExp
--- End diff --
How about move this into `UnixTime`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9165#issuecomment-155331188 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9165#issuecomment-155331108 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() (New v...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9579#issuecomment-155330800 **[Test build #45499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45499/consoleFull)** for PR 9579 at commit [`01366a6`](https://github.com/apache/spark/commit/01366a6a83ca26da6d6e1233306cf6148d5d051f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11593][SQL] Replace catalyst converter ...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9565#discussion_r44375262
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -60,924 +77,238 @@ case class ScalaUDF(
*/
- // Accessors used in genCode
- def userDefinedFunc(): AnyRef = function
- def getChildren(): Seq[Expression] = children
-
- private[this] val f = children.size match {
-case 0 =>
- val func = function.asInstanceOf[() => Any]
- (input: InternalRow) => {
-func()
- }
-
-case 1 =>
- val func = function.asInstanceOf[(Any) => Any]
- val child0 = children(0)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)))
- }
-
-case 2 =>
- val func = function.asInstanceOf[(Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)))
- }
-
-case 3 =>
- val func = function.asInstanceOf[(Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)))
- }
-
-case 4 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)),
- converter3(child3.eval(input)))
- }
-
-case 5 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- val child4 = children(4)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- lazy val converter4 =
CatalystTypeConverters.createToScalaConverter(child4.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)),
- converter3(child3.eval(input)),
- converter4(child4.eval(input)))
- }
-
-case 6 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any, Any, Any) =>
Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- val child4 = children(4)
- val child5 = children(5)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- lazy val converter4 =
CatalystTypeConverters.createToScalaConverter(child4.dataType)
- lazy val converter5 =
Ca
[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)
Github user jbonofre commented on the pull request: https://github.com/apache/spark/pull/9165#issuecomment-155330359 PR rebased and cleaned up (about indentation and style) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11593][SQL] Replace catalyst converter ...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/9565#issuecomment-155330331 @davies Thanks for reviewing. I will work on generated version later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155330108 **[Test build #45502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45502/consoleFull)** for PR 9287 at commit [`fe9de83`](https://github.com/apache/spark/commit/fe9de83ec881d8c021a4f2b790dcb05efa6a8ce0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155329879 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11618] [ML] Minor refactoring of basic ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9587#issuecomment-155329965 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11326: Split networking in standalone mo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9287#issuecomment-155329836 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11618] [ML] Minor refactoring of basic ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9587#issuecomment-155329968 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45496/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11618] [ML] Minor refactoring of basic ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9587#issuecomment-155329823 **[Test build #45496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45496/consoleFull)** for PR 9587 at commit [`7c73f62`](https://github.com/apache/spark/commit/7c73f62822abf05433d3007c7c07b185dd5d3426). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `abstract class Writer extends BaseReadWrite with Logging `\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11593][SQL] Replace catalyst converter ...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/9565#issuecomment-155329242 @viirya Thanks for work on this. I think it's more important to generate the code for converter in generated ScalaUDF. BTW, the RowEncoder is new in 1.6 (experimental feature), so I'd like to only merge this into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9490#issuecomment-155329029 **[Test build #45500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45500/consoleFull)** for PR 9490 at commit [`63ebcdf`](https://github.com/apache/spark/commit/63ebcdfd3fc251f35e85096486eaa9de77d5699c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11593][SQL] Replace catalyst converter ...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9565#discussion_r44374944
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -60,924 +77,238 @@ case class ScalaUDF(
*/
- // Accessors used in genCode
- def userDefinedFunc(): AnyRef = function
- def getChildren(): Seq[Expression] = children
-
- private[this] val f = children.size match {
-case 0 =>
- val func = function.asInstanceOf[() => Any]
- (input: InternalRow) => {
-func()
- }
-
-case 1 =>
- val func = function.asInstanceOf[(Any) => Any]
- val child0 = children(0)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)))
- }
-
-case 2 =>
- val func = function.asInstanceOf[(Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)))
- }
-
-case 3 =>
- val func = function.asInstanceOf[(Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)))
- }
-
-case 4 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)),
- converter3(child3.eval(input)))
- }
-
-case 5 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- val child4 = children(4)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- lazy val converter4 =
CatalystTypeConverters.createToScalaConverter(child4.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)),
- converter3(child3.eval(input)),
- converter4(child4.eval(input)))
- }
-
-case 6 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any, Any, Any) =>
Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- val child4 = children(4)
- val child5 = children(5)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- lazy val converter4 =
CatalystTypeConverters.createToScalaConverter(child4.dataType)
- lazy val converter5 =
Ca
[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() (New v...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9579#issuecomment-155327387 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9490#issuecomment-155327348 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9490#issuecomment-155327388 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9165#issuecomment-155327359 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() (New v...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9579#issuecomment-155327337 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9165#issuecomment-155327389 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11593][SQL] Replace catalyst converter ...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9565#discussion_r44374870
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -60,924 +77,238 @@ case class ScalaUDF(
*/
- // Accessors used in genCode
- def userDefinedFunc(): AnyRef = function
- def getChildren(): Seq[Expression] = children
-
- private[this] val f = children.size match {
-case 0 =>
- val func = function.asInstanceOf[() => Any]
- (input: InternalRow) => {
-func()
- }
-
-case 1 =>
- val func = function.asInstanceOf[(Any) => Any]
- val child0 = children(0)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)))
- }
-
-case 2 =>
- val func = function.asInstanceOf[(Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)))
- }
-
-case 3 =>
- val func = function.asInstanceOf[(Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)))
- }
-
-case 4 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)),
- converter3(child3.eval(input)))
- }
-
-case 5 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any, Any) => Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- val child4 = children(4)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- lazy val converter4 =
CatalystTypeConverters.createToScalaConverter(child4.dataType)
- (input: InternalRow) => {
-func(
- converter0(child0.eval(input)),
- converter1(child1.eval(input)),
- converter2(child2.eval(input)),
- converter3(child3.eval(input)),
- converter4(child4.eval(input)))
- }
-
-case 6 =>
- val func = function.asInstanceOf[(Any, Any, Any, Any, Any, Any) =>
Any]
- val child0 = children(0)
- val child1 = children(1)
- val child2 = children(2)
- val child3 = children(3)
- val child4 = children(4)
- val child5 = children(5)
- lazy val converter0 =
CatalystTypeConverters.createToScalaConverter(child0.dataType)
- lazy val converter1 =
CatalystTypeConverters.createToScalaConverter(child1.dataType)
- lazy val converter2 =
CatalystTypeConverters.createToScalaConverter(child2.dataType)
- lazy val converter3 =
CatalystTypeConverters.createToScalaConverter(child3.dataType)
- lazy val converter4 =
CatalystTypeConverters.createToScalaConverter(child4.dataType)
- lazy val converter5 =
Ca
[GitHub] spark pull request: [SPARK-10863][SPARKR] Method coltypes() (New v...
Github user olarayej commented on the pull request: https://github.com/apache/spark/pull/9579#issuecomment-155326781 I have addressed the docs suggestions. Thank you folks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11360] [Doc] Loss of nullability when w...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/9314#issuecomment-155326566 https://github.com/apache/spark/commit/2f38378856fb56bdd9be7ccedf56427e81701f4e --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11360] [Doc] Loss of nullability when w...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/9314#issuecomment-155326285 Sorry, I forgot to comment. It was closed because I merged it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11102] [SQL] Uninformative exception wh...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/9490#issuecomment-155325667 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11593][SQL] Replace catalyst converter ...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9565#discussion_r44374607
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
---
@@ -1041,6 +371,10 @@ case class ScalaUDF(
"""
}
- private[this] val converter =
CatalystTypeConverters.createToCatalystConverter(dataType)
- override def eval(input: InternalRow): Any = converter(f(input))
+ override def eval(input: InternalRow): Any = {
+val projected = InternalRow.fromSeq(children.map(_.eval(input)))
+val outputEncoder: ExpressionEncoder[Row] =
--- End diff --
`outputEncoder` should be created outside of `eval`, or it will be too slow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9830] [SQL] Remove AggregateExpression1...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9556#discussion_r44374094
--- Diff: R/pkg/R/functions.R ---
@@ -1339,7 +1339,7 @@ setMethod("pmod", signature(y = "Column"),
#' @export
setMethod("approxCountDistinct",
signature(x = "Column"),
- function(x, rsd = 0.95) {
+ function(x, rsd = 0.05) {
--- End diff --
Good catch, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818] Re-enable Docker tests for JDBC d...
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9503#issuecomment-155317816
Something really weird is going on here.
Following the advice at http://stackoverflow.com/a/21876380/590203, I added
the following line to `DockerJDBCIntegrationSuite` in order to figure out where
`BasicLineFormatter` was being loaded from:
```
println(this.getClass.getClassLoader.getResource("org/apache/http/message/BasicLineFormatter.class"))
```
When I run this locally with
```
sbt/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl
-Phive-thriftserver -Phive
```
I get
```
jar:file:/Users/joshrosen/Documents/spark/lib_managed/jars/httpcore-4.2.4.jar!/org/apache/http/message/BasicLineFormatter.class
```
This doesn't make sense, though: `show test:dependencyClasspath` does not
include that JAR, so I'm not quite sure how it's winding up on the classpath.
If I toss in a
```
this.getClass.getClassLoader.asInstanceOf[URLClassLoader].getURLs.foreach(println)
```
I still don't see where that JAR is coming from.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11590][SQL] use native json_tuple in la...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9562#issuecomment-155317757 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45497/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11590][SQL] use native json_tuple in la...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9562#issuecomment-155317753 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11584][SQL] The attribute of temporay t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9588#issuecomment-155317691 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11584][SQL] The attribute of temporay t...
GitHub user dilipbiswal opened a pull request: https://github.com/apache/spark/pull/9588 [SPARK-11584][SQL] The attribute of temporay table shows false The code used to hard-code the temporary attribute to false. Have changed the code to retrieve the attribute from meta store. Question - 1) I have added a test case in CliSuite as an unit test. When running the tests on jenkins, can this test get executed against older version of hive ? If so, then it will fail as the method to query the temporary attribute is only available after hive 14. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dilipbiswal/spark SPARK-11584 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9588.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9588 commit 85a20b4168c61e9439ec35e1676cc4ee79860a0d Author: Dilip Biswal Date: 2015-11-09T10:06:11Z SPARK-11584 The attribute of temporay table shows false --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Github user yu-iskw commented on the pull request: https://github.com/apache/spark/pull/5267#issuecomment-155314648 @jkbradley thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11590][SQL] use native json_tuple in la...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9562#issuecomment-155313319 **[Test build #45498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45498/consoleFull)** for PR 9562 at commit [`2764c29`](https://github.com/apache/spark/commit/2764c298338c4f5024a3481df9c81488939bbbf3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11590][SQL] use native json_tuple in la...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9562#issuecomment-155312651 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11590][SQL] use native json_tuple in la...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9562#issuecomment-155312760 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
