[GitHub] spark issue #14437: fix 2 typos
Github user wongxingjun commented on the issue: https://github.com/apache/spark/pull/14437 Yeah well that might be my OCD. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14406: [SPARK-16778][SQL][TRIVIAL] Fix deprecation warni...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14406 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14416 Merged to master/2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14380: [SPARK-16485][DOC][ML] Remove useless latex in a ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14380 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14406: [SPARK-16778][SQL][TRIVIAL] Fix deprecation warning with...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14406 Merged to master/2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14380: [SPARK-16485][DOC][ML] Remove useless latex in a log mes...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14380 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14420 I'll wait a moment to see if @JoshRosen has concerns about it. The original change was approved, for branch 1.6, but in re-reading, I wonder if it was only accidentally discussed as 1.6-only. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14432: [SPARK-16765] [Example] Added example for k-means to pre...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14432 This is WontFix per the JIRA, please close the PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14436: [SPARK-16816] Modify java example which is also reflect ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14436 OK, it doesn't hurt. OK to merge if this seems to help your case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14438: [SPARK-16822][DOC] Support latex in scaladoc.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14438 **[Test build #63077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63077/consoleFull)** for PR 14438 at commit [`2a480ca`](https://github.com/apache/spark/commit/2a480caba91dd912eeda8b2b4393f9be11061310). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14437: fix 2 typos
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14437 I think this is too trivial to bother with. The structure isn't parallel but the fragment isn't exactly ungrammatical --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14438: [SPARK-16822][DOC] Support latex in scaladoc.
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14438 Nice, does it work? :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14438: [SPARK-16822][DOC] Support latex in scaladoc.
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14438 Jenkins add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14438: [SPARK-16822][DOC] Support latex in scaladoc.
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14438 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14411: [SPARK-16804][SQL] Correlated subqueries containing LIMI...
Github user nsyca commented on the issue: https://github.com/apache/spark/pull/14411 @hvanhovell, Thank you for your comment. There are quite a few patterns being blacklisted already, such as correlation under set operators (UNION, EXCEPT, INTERSECT), correlation outside of WHERE/HAVING context, correlation in the right table of a LEFT [OUTER] JOIN (and the left table of a RIGHT [OUTER] JOIN]). I am working on discovering more issues in this area but it looks like a bigger project to me. I have a general idea that the rewrite of correlation subquery to join should not happen in the Analysis phase. We should build a Logical plan to represent the subquery and perform the rewrite at the Optimizer phase instead. I am new to the Spark code and this is my first PR. So I'd like to make it a small, self-contained project to gain my confidence in working with the code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14438: [SPARK-16822][DOC] Support latex in scaladoc.
GitHub user lins05 opened a pull request: https://github.com/apache/spark/pull/14438 [SPARK-16822][DOC] Support latex in scaladoc. ## What changes were proposed in this pull request? Support using latex in scaladoc by adding MathJax javascript to the js template. ## How was this patch tested? Generated scaladoc. Preview: - LogisticGradient: [before](https://spark.apache.org/docs/2.0.0/api/scala/index.html#org.apache.spark.mllib.optimization.LogisticGradient) and [after](https://sparkdocs.lins05.pw/spark-16822/api/scala/index.html#org.apache.spark.mllib.optimization.LogisticGradient) - MinMaxScaler: [before](https://spark.apache.org/docs/2.0.0/api/scala/index.html#org.apache.spark.ml.feature.MinMaxScaler) and [after](https://sparkdocs.lins05.pw/spark-16822/api/scala/index.html#org.apache.spark.ml.feature.MinMaxScaler) You can merge this pull request into a Git repository by running: $ git pull https://github.com/lins05/spark spark-16822-support-latex-in-scaladoc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14438.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14438 commit 2a480caba91dd912eeda8b2b4393f9be11061310 Author: Shuai Lin Date: 2016-08-01T08:06:49Z [SPARK-16822][DOC] Support latex in scaladoc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12896: [SPARK-14489][ML][PYSPARK] ALS unknown user/item predict...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/12896 Hm, let me go through the logic with you one more time here. Isn't it better in theory to fix the model to not return NaN, but rather return _some_ default answer, even if it's "0" or equivalent? this is at least no worse for scoring, and fixes the evaluation problem. New users and items are reasonable conditions for this model, not an error case. The current behavior isn't that helpful, so I'm not sure leaving it as a choice is doing anybody a favor. My concern with the "drop" mode is that it is not penalizing any case where the model can't make an answer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14438: [SPARK-16822][DOC] Support latex in scaladoc.
Github user lins05 commented on the issue: https://github.com/apache/spark/pull/14438 @srowen could you take look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14438: [SPARK-16822][DOC] Support latex in scaladoc.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14438 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14400 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14400 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63076/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14400 **[Test build #63076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63076/consoleFull)** for PR 14400 at commit [`e0549a9`](https://github.com/apache/spark/commit/e0549a998f3d75eeedf4334e9d61ad57dbe52efe). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14212: [SPARK-16558][Examples][MLlib] examples/mllib/LDAExample...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/14212 The fix LGTM - I'm in two minds about whether this example is "good" (illustrating using both `ml` and `mllib` APIs together) or "bad" (for the same reason, since there is LDA available in `ml`). And we already have a `LatentDirichletExample` in `mllib`... seems to me that these `mllib` examples could use a clean up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #11157: [SPARK-11714][Mesos] Make Spark on Mesos honor po...
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11157#discussion_r72971469
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
---
@@ -423,18 +432,35 @@ private[spark] class
MesosCoarseGrainedSchedulerBackend(
tasks.toMap
}
+ private def getResources(resources: JList[Resource], taskCPUs: Int,
taskMemory: Int)
+: (List[Resource], List[List[Resource]], List[Long]) = {
+val (afterCPUResources, cpuResourcesToUse) =
+ partitionResources(resources, "cpus", taskCPUs)
+val (afterMemResources, memResourcesToUse) =
+ partitionResources(afterCPUResources.asJava, "mem", taskMemory)
--- End diff --
ok np
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #11157: [SPARK-11714][Mesos] Make Spark on Mesos honor po...
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11157#discussion_r72971304
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
---
@@ -423,18 +432,35 @@ private[spark] class
MesosCoarseGrainedSchedulerBackend(
tasks.toMap
}
+ private def getResources(resources: JList[Resource], taskCPUs: Int,
taskMemory: Int)
+: (List[Resource], List[List[Resource]], List[Long]) = {
+val (afterCPUResources, cpuResourcesToUse) =
+ partitionResources(resources, "cpus", taskCPUs)
+val (afterMemResources, memResourcesToUse) =
+ partitionResources(afterCPUResources.asJava, "mem", taskMemory)
+// process port offers
+val (resourcesWithoutPorts, portResources) =
filterPortResources(afterMemResources)
+val (afterPortResources, portResourcesToUse, portsToUse) =
+ partitionPorts(conf, portResources)
+(resourcesWithoutPorts ++ afterPortResources,
--- End diff --
ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14437: fix 2 typos
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14437 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14437: fix 2 typos
GitHub user wongxingjun opened a pull request: https://github.com/apache/spark/pull/14437 fix 2 typos ## What changes were proposed in this pull request? 2 typos are fixed. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) You can merge this pull request into a Git repository by running: $ git pull https://github.com/wongxingjun/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14437.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14437 commit 0cac36609a146db7bc2c59c27cf1b5e1d392c6e1 Author: Xingjun Wang Date: 2016-08-01T12:30:24Z fix 2 typos --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11157: [SPARK-11714][Mesos] Make Spark on Mesos honor port rest...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11157 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63075/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11157: [SPARK-11714][Mesos] Make Spark on Mesos honor port rest...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11157 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11157: [SPARK-11714][Mesos] Make Spark on Mesos honor port rest...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11157 **[Test build #63075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63075/consoleFull)** for PR 11157 at commit [`2493d2a`](https://github.com/apache/spark/commit/2493d2a1a5a4429f6b16d3767d7a026893445745). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14400 we have an individual style check script: `./dev/scalastyle` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...
Github user eyalfa commented on the issue: https://github.com/apache/spark/pull/14400 @cloud-fan, just for notice for the next patches: is there a way to run mvn in a mode that fails on style issues? it could save a lot of round-trips if it existed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14400 **[Test build #63076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63076/consoleFull)** for PR 14400 at commit [`e0549a9`](https://github.com/apache/spark/commit/e0549a998f3d75eeedf4334e9d61ad57dbe52efe). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14400 https://github.com/apache/spark/pull/14400#discussion_r72897142 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14400: [Spark-16791] [SQL] cast struct with timestamp field fai...
Github user eyalfa commented on the issue: https://github.com/apache/spark/pull/14400 @cloud-fan, any update on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11157: [SPARK-11714][Mesos] Make Spark on Mesos honor port rest...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11157 **[Test build #63075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63075/consoleFull)** for PR 11157 at commit [`2493d2a`](https://github.com/apache/spark/commit/2493d2a1a5a4429f6b16d3767d7a026893445745). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14258: [Spark-16579][SparkR] add install.spark function
Github user junyangq commented on a diff in the pull request:
https://github.com/apache/spark/pull/14258#discussion_r72950957
--- Diff: R/pkg/R/install.R ---
@@ -36,7 +36,7 @@
#' \code{without-hadoop}.
#'
#' @param hadoopVersion Version of Hadoop to install. Default is
\code{"2.7"}. It can take other
-#' version number in the format of "int.int".
+#' version number in the format of "x.y" where x and
y are integer.
--- End diff --
Yes, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14258: [Spark-16579][SparkR] add install.spark function
Github user junyangq commented on a diff in the pull request:
https://github.com/apache/spark/pull/14258#discussion_r72950905
--- Diff: R/pkg/R/install.R ---
@@ -0,0 +1,232 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Functions to install Spark in case the user directly downloads SparkR
+# from CRAN.
+
+#' Download and Install Apache Spark to a Local Directory
+#'
+#' \code{install.spark} downloads and installs Spark to a local directory
if
+#' it is not found. The Spark version we use is the same as the SparkR
version.
+#' Users can specify a desired Hadoop version, the remote mirror site, and
+#' the directory where the package is installed locally.
+#'
+#' The full url of remote file is inferred from \code{mirrorUrl} and
\code{hadoopVersion}.
+#' \code{mirrorUrl} specifies the remote path to a Spark folder. It is
followed by a subfolder
+#' named after the Spark version (that corresponds to SparkR), and then
the tar filename.
+#' The filename is composed of four parts, i.e. [Spark
version]-bin-[Hadoop version].tgz.
+#' For example, the full path for a Spark 2.0.0 package for Hadoop 2.7 from
+#' \code{http://apache.osuosl.org} has path:
+#'
\code{http://apache.osuosl.org/spark/spark-2.0.0/spark-2.0.0-bin-hadoop2.7.tgz}.
+#' For \code{hadoopVersion = "without"}, [Hadoop version] in the filename
is then
+#' \code{without-hadoop}.
+#'
+#' @param hadoopVersion Version of Hadoop to install. Default is
\code{"2.7"}. It can take other
+#' version number in the format of "int.int".
+#' If \code{hadoopVersion = "without"}, "Hadoop free"
build is installed.
+#' See
+#'
\href{http://spark.apache.org/docs/latest/hadoop-provided.html}{
+#' "Hadoop Free" Build} for more information.
+#' Other patched version names can also be used, e.g.
\code{"cdh4"}
+#' @param mirrorUrl base URL of the repositories to use. The directory
layout should follow
+#'
\href{http://www.apache.org/dyn/closer.lua/spark/}{Apache mirrors}.
+#' @param localDir a local directory where Spark is installed. The
directory contains
+#' version-specific folders of Spark packages. Default is
path to
+#' the cache directory:
+#' \itemize{
+#' \item Mac OS X: \file{~/Library/Caches/spark}
+#' \item Unix: \env{$XDG_CACHE_HOME} if defined,
otherwise \file{~/.cache/spark}
+#' \item Win XP:
+#' \file{C:\\Documents and
SettingsLocal Settings\\Application
+#' Data\\spark\\spark\\Cache}
+#' \item Win Vista:
+#'
\file{C:\\UsersAppData\\Local\\spark\\spark\\Cache}
+#' }
+#' @param overwrite If \code{TRUE}, download and overwrite the existing
tar file in localDir
+#' and force re-install Spark (in case the local
directory or file is corrupted)
+#' @return \code{install.spark} returns the local directory where Spark is
found or installed
+#' @rdname install.spark
+#' @name install.spark
+#' @export
+#' @examples
+#'\dontrun{
+#' install.spark()
+#'}
+#' @note install.spark since 2.1.0
+#' @seealso See available Hadoop versions:
+#' \href{http://spark.apache.org/downloads.html}{Apache Spark}
+install.spark <- function(hadoopVersion = "2.7", mirrorUrl = NULL,
+ localDir = NULL, overwrite = FALSE) {
+ version <- paste0("spark-", packageVersion("SparkR"))
+ hadoopVersion <- tolower(hadoopVersion)
+ hadoopVersionName <- hadoop_version_name(hadoopVersion)
+ packageName <- paste(version, "bin", hadoopVersionName, sep = "-")
+ localDir <- ifelse(is.null(localDir), spark_cache_path(),
+ normalizePath(localDir, mustWork = FALSE))
+
+ if (is.na(f
[GitHub] spark issue #14435: [SPARK-16756][SQL][WIP] Add `sql` function to LogicalPla...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14435 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14435: [SPARK-16756][SQL][WIP] Add `sql` function to LogicalPla...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14435 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63074/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14435: [SPARK-16756][SQL][WIP] Add `sql` function to LogicalPla...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14435 **[Test build #63074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63074/consoleFull)** for PR 14435 at commit [`1392c8a`](https://github.com/apache/spark/commit/1392c8a62e57c0a9b66555d4ac676eb0269533a3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait NonSQLPlan extends LogicalPlan ` * `case class ReturnAnswer(child: LogicalPlan) extends UnaryNode with NonSQLPlan ` * `case class With(child: LogicalPlan, cteRelations: Map[String, SubqueryAlias])` * `case class StreamingExecutionRelation(source: Source, output: Seq[Attribute])` * `case class MemoryPlan(sink: MemorySink, output: Seq[Attribute]) extends LeafNode with NonSQLPlan ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14391: [SPARK-16767][SQL] Add existsRecursively to UserDefinedT...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14391 cc @mengxr what's the plan for UDT? Is it a public API now? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14436: [SPARK-16816] Modify java example which is also refelect...
Github user phalodi commented on the issue: https://github.com/apache/spark/pull/14436 @srowen @rxin Hey guyss i make changes as you suggested but yeah it not exactly what you suggested, i change in example which is also reflect in documentation, so please review it and give give your comments i will change them if you guyss think its not look good. Changes which i made  In currently spark 2.0 example for Json Dataset  In previous version example is as follows.  So i make 2.0 example also same as previous version while user do this he/she can also know how to create JavaSparkContext by this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14436: [SPARK-16816] Modify java example which is also refelect...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14436 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14436: modify java example which is also refelect in doc...
GitHub user phalodi opened a pull request: https://github.com/apache/spark/pull/14436 modify java example which is also refelect in documentation exmaple ## What changes were proposed in this pull request? Modify java example which is also reflect in document. ## How was this patch tested? run test cases. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phalodi/spark SPARK-16816 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14436.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14436 commit 3461517386ac01c87e8647e191013a092c4f87bd Author: sandy Date: 2016-08-01T09:19:26Z modify java example which is also refelect in documentation exmaple --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14363 Well, Spark SQL is not announced to be fully compatible with hive, I think it's reasonable to have some issues. cc @rxin @yhuai should we fix this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14355: [SPARK-16726][SQL] Improve `Union/Intersect/Excep...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14355 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14355: [SPARK-16726][SQL] Improve `Union/Intersect/Except` erro...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14355 LGTM - merging to master. Thanks for working on this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14155: [SPARK-16498][SQL] move hive hack for data source table ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14155 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14155: [SPARK-16498][SQL] move hive hack for data source table ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14155 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63073/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14155: [SPARK-16498][SQL] move hive hack for data source table ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14155 **[Test build #63073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63073/consoleFull)** for PR 14155 at commit [`9ae7a71`](https://github.com/apache/spark/commit/9ae7a71c0d04aa58fd221b2bf084cf3a0e8f3168). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging with Serializable ` * `case class MonotonicallyIncreasingID() extends LeafExpression with Nondeterministic ` * `case class SparkPartitionID() extends LeafExpression with Nondeterministic ` * `case class AggregateExpression(` * `case class CurrentDatabase() extends LeafExpression with Unevaluable ` * `class GenericInternalRow(val values: Array[Any]) extends BaseGenericInternalRow ` * `class AbstractScalaRowIterator[T] extends Iterator[T] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...
Github user lianhuiwang commented on the issue:
https://github.com/apache/spark/pull/14363
@cloud-fan There is a case that i met. The varchar(length)/char(length)
type is not a String Type. But now SparkSQL consider them a string type. So
there are different result with the following example:
TestHive.sessionState.metadataHive.runSqlHive("CREATE TABLE test (id
varchar(50))")
TestHive.sessionState.metadataHive.runSqlHive("INSERT INTO TABLE test
VALUES ('abcdef')")
TestHive.sessionState.metadataHive.runSqlHive("CREATE TABLE test_parquet
(id varchar(2) stored as parquet)")
TestHive.sessionState.metadataHive.runSqlHive("insert overwrite table
varchar_parquet1 select * from test")
the result of varchar_parquet1 are 'ab'.
spark.sql("insert overwrite table varchar_parquet1 select * from
test").show()
the result of varchar_parquet1 are 'abcdef'.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14435: [SPARK-16756][SQL][WIP] Add `sql` function to LogicalPla...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14435 **[Test build #63074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63074/consoleFull)** for PR 14435 at commit [`1392c8a`](https://github.com/apache/spark/commit/1392c8a62e57c0a9b66555d4ac676eb0269533a3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14435: [SPARK-16756][SQL][WIP] Add `sql` function to Log...
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/14435 [SPARK-16756][SQL][WIP] Add `sql` function to LogicalPlan and `NonSQLPlan` trait ## What changes were proposed in this pull request? This PR is a part of [SPARK-16576](https://issues.apache.org/jira/browse/SPARK-16576) that moves logicalPlan SQL generation code from SQLBuilder into logical operators. Like `Expression`, this PR adds `sql` function for `LogicalPlan` and `NonSQLPlan` trait. The method will be `abstract` method. All logical plan should implement that or use trait `NonSQLPlan` explicitly. ```scala /** * Returns SQL representation of this plan. For the plans extending [[NonSQLPlan]], * this method may return an arbitrary user facing string. */ def sql: String ``` This PR updates testsuites including`LogicalPlanToSQLSuite` and `ExpressionToSQLSuite` in order to test new `sql` function, but does not remove `SQLBuilder` and its usage in `views.scala` of `sql/core`. ## How was this patch tested? Pass the Jenkins tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dongjoon-hyun/spark SPARK-16756 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14435.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14435 commit 1392c8a62e57c0a9b66555d4ac676eb0269533a3 Author: Dongjoon Hyun Date: 2016-08-01T07:00:11Z [SPARK-16756][SQL] Add `sql` function to LogicalPlan and `NonSQLPlan` trait --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14434: [SPARK-16828][SQL] remove MaxOf and MinOf
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14434 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63072/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14434: [SPARK-16828][SQL] remove MaxOf and MinOf
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14434 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14434: [SPARK-16828][SQL] remove MaxOf and MinOf
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14434 **[Test build #63072 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63072/consoleFull)** for PR 14434 at commit [`ce0b9fe`](https://github.com/apache/spark/commit/ce0b9fee5006e0dc89869e25056ccaa72db972e5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11673: [SPARK-13853][SQL] QueryPlan sub-classes should override...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11673 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63070/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11673: [SPARK-13853][SQL] QueryPlan sub-classes should override...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11673 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11673: [SPARK-13853][SQL] QueryPlan sub-classes should override...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11673 **[Test build #63070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63070/consoleFull)** for PR 11673 at commit [`ce0b9fe`](https://github.com/apache/spark/commit/ce0b9fee5006e0dc89869e25056ccaa72db972e5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14401: [SPARK-16793][SQL]Set the temporary warehouse path to sc...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/14401 cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11601: [SPARK-13568] [ML] Create feature transformer to impute ...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/11601 @hhbyyh could you update the since annotations to target `2.1.0`? @jkbradley if you have a chance to review would be great. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14118: [SPARK-16462][SPARK-16460][SPARK-15144][SQL] Make CSV ca...
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/14118 Here are some findings as I dug a little: 1. Since https://github.com/databricks/spark-csv/pull/102(Jul, 2015), we would cast `""` as `null` for all types other than strings. For strings, `""` would still be `""`; 2. Then we had added `treatEmptyValuesAsNulls` in https://github.com/databricks/spark-csv/pull/147(Sep, 2015), after which, `""` would be `null` when `treatEmptyValuesAsNulls == true` and would be still `""` otherwise; 3. Then we had added `nullValue` in https://github.com/databricks/spark-csv/pull/224(Dec, 2015), so people could specify some string like `"MISSING"` other than the default `""` to represent null values. Then after the above 1.2.3., we have the following, which seems reasonable and is backward-compatible: (default) when nullVale == "" when nullValue == "MISSING" (default) when treatEmptyValuesAsNulls == false "" would cast to "" "" would cast to "" when treatEmptyValuesAsNulls == true "" would cast to null "" would cast to "" However we don't have this `treatEmptyValuesAsNulls` in Spark 2.0. @falaki would it be OK with you if I add it back? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12896: [SPARK-14489][ML][PYSPARK] ALS unknown user/item predict...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/12896 cc also @xsankar who was interested on the related JIRAs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14363 Your concern is valid. We are missing the test cases for verifying these scenarios. I saw a discussion in a wechat group about the issue in integration between Hive and Spark. They are complaining Spark is unable to read the data wrote by Hive. In Hive refactoring, I am wondering if we also need to build the test cases to cover these cases? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #12896: [SPARK-14489][ML][PYSPARK] ALS unknown user/item ...
Github user MLnick commented on a diff in the pull request:
https://github.com/apache/spark/pull/12896#discussion_r72931611
--- Diff: python/pyspark/ml/recommendation.py ---
@@ -332,6 +338,20 @@ def getFinalStorageLevel(self):
"""
return self.getOrDefault(self.finalStorageLevel)
+@since("2.0.0")
--- End diff --
Ah yeah I totally forgot to update these. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
