[GitHub] spark issue #15935: [SPARK-18188] add checksum for blocks of broadcast
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15935 LGTM. Merging to master and 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15935: [SPARK-18188] add checksum for blocks of broadcas...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15935 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/15975#discussion_r89956606 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -76,9 +76,6 @@ class JDBCOptions( // the number of partitions val numPartitions = parameters.get(JDBC_NUM_PARTITIONS).map(_.toInt) --- End diff -- Reading the table using a single partition. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15975#discussion_r89956633
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
---
@@ -667,13 +667,13 @@ object JdbcUtils extends Logging {
val getConnection: () => Connection = createConnectionFactory(options)
val batchSize = options.batchSize
val isolationLevel = options.isolationLevel
-val numPartitions = options.numPartitions
-val repartitionedDF =
- if (numPartitions.isDefined && numPartitions.get <
df.rdd.getNumPartitions) {
-df.coalesce(numPartitions.get)
- } else {
-df
- }
+val repartitionedDF = options.numPartitions match {
+ case Some(n) if n <= 0 => throw new IllegalArgumentException(
--- End diff --
Yeah.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15975#discussion_r89956695
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -209,6 +209,14 @@ class JDBCSuite extends SparkFunSuite
conn.close()
}
+ // Check whether the tables are fetched in the expected degree of
parallelism
+ def checkNumPartitions(df: DataFrame, expectedNumPartitions: Int): Unit
= {
+val explain = ExplainCommand(df.queryExecution.logical, extended =
true)
+val plans = spark.sessionState.executePlan(explain).executedPlan
+val expectedMsg =
s"${JDBCOptions.JDBC_NUM_PARTITIONS}=$expectedNumPartitions"
--- End diff --
Good idea!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16030 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69307/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16030 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15954: [WIP][SPARK-18516][SQL] Split state and progress in stre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15954 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto pick" fe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16052 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15954: [WIP][SPARK-18516][SQL] Split state and progress in stre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15954 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16048 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69309/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16045: [SPARK-18553][CORE] Fix leak of TaskSetManager following...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16045 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16017: [SPARK-18592][ML] Move DT/RF/GBT Param setter methods to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16017 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69306/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16048 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16028 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69305/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16017: [SPARK-18592][ML] Move DT/RF/GBT Param setter methods to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16017 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto pick" fe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16052 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69303/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15954: [WIP][SPARK-18516][SQL] Split state and progress in stre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15954 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69295/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16045: [SPARK-18553][CORE] Fix leak of TaskSetManager following...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16045 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69304/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15954: [WIP][SPARK-18516][SQL] Split state and progress in stre...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15954 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69308/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16028 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16055: [SPARK-17897] [SQL] Attribute is not NullIntolera...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/16055
[SPARK-17897] [SQL] Attribute is not NullIntolerant
### What changes were proposed in this pull request?
`Attribute` is not `NullIntolerant`. This PR is to fix it.
Without the fix, the following test case will return empty.
```Scala
val data = Seq[java.lang.Integer](1, null).toDF("key")
data.filter("not key is not null").show()
```
### How was this patch tested?
Added a test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark isNotNull
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16055.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16055
commit 33c10a0994c9802df901f211e1f28c52e34df27f
Author: gatorsmile
Date: 2016-11-29T08:00:55Z
fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16055: [SPARK-17897] [SQL] Attribute is not NullIntolerant
Github user rxin commented on the issue: https://github.com/apache/spark/pull/16055 Can you explain how did nullintolerant impact the case? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16017: [SPARK-18592][ML] Move DT/RF/GBT Param setter methods to...
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16017 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16030 **[Test build #69312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69312/consoleFull)** for PR 16030 at commit [`43f028d`](https://github.com/apache/spark/commit/43f028d3b495a825cebff39daa12d6a2f25f0110). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16055: [SPARK-17897] [SQL] Attribute is not NullIntolerant
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16055 **[Test build #69310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69310/consoleFull)** for PR 16055 at commit [`33c10a0`](https://github.com/apache/spark/commit/33c10a0994c9802df901f211e1f28c52e34df27f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16048 **[Test build #69311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69311/consoleFull)** for PR 16048 at commit [`fa313e5`](https://github.com/apache/spark/commit/fa313e5fdd60783df6fb96403802d4d1558b8cca). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16055: [SPARK-17897] [SQL] Attribute is not NullIntolerant
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16055 Sure, will update the PR description tomorrow. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16017: [SPARK-18592][ML] Move DT/RF/GBT Param setter methods to...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16017 **[Test build #69313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69313/consoleFull)** for PR 16017 at commit [`30f5096`](https://github.com/apache/spark/commit/30f5096ce9dce89e3d3a3014bc53164cc2af2788). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/16014
@shivaram Cool, I did know about release-build but I didn't know it's
running on Jenkins. I *think* we should be ok but might want to check Jenkins
has "e1071" and "survival" which are optional for compatibility tests but `R
CMD check` is enforcing/requiring it. If you recall,
[this](https://github.com/apache/spark/pull/15790#issuecomment-259780799) is
this conversation that prompted this change.
@rxin This PR updates what goes into the Spark binary release to match what
we (intend to) release on CRAN for the R package
As for the diff, this is the delta between this PR and Spark 2.0.2 under
the R/lib/SparkR directory. It turns out `R CMD check` also depends on Rd file
generation in install-dev.sh (ie. `devtools::document(pkg="./pkg",
roclets=c("rd")) }`).. this is going to take more time to untangle this in a
follow up.
_what's additional_
```
SparkR/
-rw-r--r-- INDEX
drwxr-xr-x doc
SparkR/Meta/
-rw-r--r-- vignette.rds
SparkR/doc/
-rw-r--r-- sparkr-vignettes.Rmd
-rw-r--r-- sparkr-vignettes.R
-rw-r--r-- sparkr-vignettes.html
-rw-r--r-- index.html
```
_what's omitted_
```
SparkR/html/
-rw-r--r-- 1 root root 1319 Nov 29 08:05 R.css
-rw-r--r-- 1 root root 81153 Nov 29 08:05 00Index.html
```
What it used to have `year.html` `write.parquet.html`
`sparkR.session.html`, the html directory now only has 2 files. My
understanding is these knitr html output are actually not used at runtime. I
checked that `?sparkR.session` in the `sparkR` is still working correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user zhengruifeng commented on the issue: https://github.com/apache/spark/pull/16028 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16028 **[Test build #69314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69314/consoleFull)** for PR 16028 at commit [`74cb363`](https://github.com/apache/spark/commit/74cb3639278fa525dfe5b75d11a7a8dcd06f04a3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69315/consoleFull)** for PR 16014 at commit [`c9c9802`](https://github.com/apache/spark/commit/c9c9802e67178e2283b1ebc9fa13f39db916773d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs,...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16009 **[Test build #69316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69316/consoleFull)** for PR 16009 at commit [`019e5af`](https://github.com/apache/spark/commit/019e5afc4e50ba6910734c1a9ac60df13065f6a2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scal...
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/16009#discussion_r89961315
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala ---
@@ -49,15 +49,13 @@ private[feature] trait ChiSqSelectorParams extends
Params
*
* @group param
*/
- @Since("1.6.0")
--- End diff --
Yeah, theoretically we should do that, but I'm not very confidence whether
this change is appropriate. If we meet an agreement on how to deal with this
issue, we can address other places in this PR or follow-up work. cc @jkbradley
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scal...
Github user yanboliang commented on a diff in the pull request: https://github.com/apache/spark/pull/16009#discussion_r89961575 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala --- @@ -70,17 +70,16 @@ private[feature] trait QuantileDiscretizerBase extends Params * invalid values), error (throw an error), or keep (keep invalid values in a special additional * bucket). * Default: "error" + * TODO: Reuse handleInvalid in HasHandleInvalid. --- End diff -- Yeah, updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16045: [SPARK-18553][CORE] Fix leak of TaskSetManager following...
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/16045 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16044: [Spark-18614][SQL] Incorrect predicate pushdown f...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16044#discussion_r89962468
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala ---
@@ -575,6 +575,24 @@ class JoinSuite extends QueryTest with
SharedSQLContext {
Row(3, 2) :: Nil)
}
+ test("predicate push down in left anti join") {
+val df = sql("SELECT l.a, l.b FROM testData2 l LEFT ANTI JOIN testData
r ON l.a = l.b")
--- End diff --
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16045: [SPARK-18553][CORE] Fix leak of TaskSetManager following...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16045 **[Test build #69317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69317/consoleFull)** for PR 16045 at commit [`9689763`](https://github.com/apache/spark/commit/96897631dfb5efc9cbf94044ea4eb0a8764ae78c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16017: [SPARK-18592][ML] Move DT/RF/GBT Param setter methods to...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16017 **[Test build #69313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69313/consoleFull)** for PR 16017 at commit [`30f5096`](https://github.com/apache/spark/commit/30f5096ce9dce89e3d3a3014bc53164cc2af2788). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16017: [SPARK-18592][ML] Move DT/RF/GBT Param setter methods to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16017 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69313/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16017: [SPARK-18592][ML] Move DT/RF/GBT Param setter methods to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16017 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16054: dstreaming receiver for twitter
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16054 It seems like you didn't read http://spark.apache.org/contributing.html -- it's even left in the message you didn't delete above. This isn't how changes are proposed. Twitter integration moved to Bahir anyway: http://bahir.apache.org/docs/spark/current/spark-streaming-twitter/ Close this please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16002: [SPARK-18341][ML] Eliminate use of SingularMatrixExcepti...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16002 Yes, echoing @sethah, can I ask what the problem being solved is? I understand being careful with what exactly is thrown from _public_ methods because they end up forming part of the API, but this seems to be about internal-only APIs. I have heard the mantra "don't use exceptions for control flow" but it's wrong. They are certainly a valid way to signal an out-of-band result from a method. In fact, it's a mechanism with its own language support. Reinventing exception handling without exceptions using error codes is just using "exceptions" for control flow but without actual making use of the simple mechanism for it. It's true that making an exception can be non-trivially expensive because it must fill in a stack trace, but that's no issue here (and can be suppressed if needed) Of course there are ways to abuse any language mechanism, and cases where exceptions shouldn't be used, but exceptional cases like "the matrix is singular" are what exceptions are for. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16029: [MINOR][ML] Fix wrong @since version in KMeans
Github user zhengruifeng closed the pull request at: https://github.com/apache/spark/pull/16029 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69314/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16028 **[Test build #69314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69314/consoleFull)** for PR 16028 at commit [`74cb363`](https://github.com/apache/spark/commit/74cb3639278fa525dfe5b75d11a7a8dcd06f04a3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16028 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16030 **[Test build #69312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69312/consoleFull)** for PR 16030 at commit [`43f028d`](https://github.com/apache/spark/commit/43f028d3b495a825cebff39daa12d6a2f25f0110). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16030 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69312/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16030 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16013: [SPARK-3359][DOCS] Make javadoc8 working for unidoc/genj...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16013 Merged to master/2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16013: [SPARK-3359][DOCS] Make javadoc8 working for unid...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16013 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs,...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16009 **[Test build #69316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69316/consoleFull)** for PR 16009 at commit [`019e5af`](https://github.com/apache/spark/commit/019e5afc4e50ba6910734c1a9ac60df13065f6a2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs,...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16009 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs,...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16009 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69316/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16028 **[Test build #69318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69318/consoleFull)** for PR 16028 at commit [`bad1735`](https://github.com/apache/spark/commit/bad173563636eb55a2960e37be450f7a127fb0ee). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16056: [SPARK-18623][SQL] Add `returnNullable` to `Stati...
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/16056 [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke` and modify it to handle properly. ## What changes were proposed in this pull request? Add `returnNullable` to `StaticInvoke` the same as #15780 is trying to add to `Invoke` and modify to handle properly. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ueshin/apache-spark issues/SPARK-18623 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16056.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16056 commit 886beb0c5de8d2a971937200bd57f1f21cee253d Author: Takuya UESHIN Date: 2016-11-29T04:20:22Z Add `returnNullable` to `StaticInvoke` and modify it to handle properly. commit ab9d6fcdcc3ba82c0fe4e86add47355e7fe759e6 Author: Takuya UESHIN Date: 2016-11-29T05:36:35Z Add `returnNullable` parameter to callers of `StaticInvoke`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16054: dstreaming receiver for twitter
Github user Aegeaner closed the pull request at: https://github.com/apache/spark/pull/16054 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16055: [SPARK-17897] [SQL] Attribute is not NullIntolerant
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16055 **[Test build #69310 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69310/consoleFull)** for PR 16055 at commit [`33c10a0`](https://github.com/apache/spark/commit/33c10a0994c9802df901f211e1f28c52e34df27f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class Attribute extends LeafExpression with NamedExpression ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16055: [SPARK-17897] [SQL] Attribute is not NullIntolerant
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16055 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69310/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16055: [SPARK-17897] [SQL] Attribute is not NullIntolerant
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16055 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16056 **[Test build #69319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69319/consoleFull)** for PR 16056 at commit [`ab9d6fc`](https://github.com/apache/spark/commit/ab9d6fcdcc3ba82c0fe4e86add47355e7fe759e6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15837: [SPARK-18395][SQL] Evaluate common subexpression like la...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/15837 IMHO, it would be good to stop applying subexpression elimination to a very simple expression (e.g. `Literal(1))`). This is because it takes some cost (e.g. executing a conditional branch) for this subexpression elimination at runtime. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15736: [SPARK-18224] [CORE] Optimise PartitionedPairBuffer impl...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15736 If that's true then again doesn't my suggestion to inline `partitionComparator` fix it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16057: [SPARK-18624][SQL] Implicit cast complex types
GitHub user jiangxb1987 opened a pull request: https://github.com/apache/spark/pull/16057 [SPARK-18624][SQL] Implicit cast complex types ## What changes were proposed in this pull request? Currently `ImplicitTypeCasts` doesn't handle casts between `ArrayType`s, this is not convenient, we should add a rule to enable casting from `ArrayType(InternalType)` to `ArrayType(newInternalType)`. Goals: 1. Add a rule to `ImplicitTypeCasts` to enable casting between `ArrayType`s; 2. Simplify `Percentile` and `ApproximatePercentile`. ## How was this patch tested? Updated test cases in `TypeCoercionSuite`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jiangxb1987/spark implicit-cast-complex-types Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16057.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16057 commit e4610109f7b0559863122c8d01d470b08d462f86 Author: jiangxingbo Date: 2016-11-29T09:42:39Z Support implict type casts between ArrayTypes. commit 374b4f82bb59ef4ea3279c22dd6e9cd22f710ec9 Author: jiangxingbo Date: 2016-11-29T10:10:55Z update Percentile and ApproximatePercentile. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16058: [SPARK-18291][SparkR][ML][FOLLOW-UP] Encode proba...
GitHub user yanboliang opened a pull request: https://github.com/apache/spark/pull/16058 [SPARK-18291][SparkR][ML][FOLLOW-UP] Encode probability to prediction by SQLTransformer. ## What changes were proposed in this pull request? This is follow-up for #15788, ```spark.glm``` can encode probability to prediction by ```SQLTransformer``` rather than reimplement a new transformer. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yanboliang/spark spark-18291-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16058.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16058 commit 7644e977fb280dd493f581fb5b07c2a0aaa2254c Author: Yanbo Liang Date: 2016-11-29T10:18:31Z Encode probability to prediction by SQLTransformer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16058: [SPARK-18291][SparkR][ML][FOLLOW-UP] Encode probability ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16058 **[Test build #69320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69320/consoleFull)** for PR 16058 at commit [`7644e97`](https://github.com/apache/spark/commit/7644e977fb280dd493f581fb5b07c2a0aaa2254c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16057: [SPARK-18624][SQL] Implicit cast complex types
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16057 **[Test build #69321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69321/consoleFull)** for PR 16057 at commit [`374b4f8`](https://github.com/apache/spark/commit/374b4f82bb59ef4ea3279c22dd6e9cd22f710ec9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16058: [SPARK-18291][SparkR][ML][FOLLOW-UP] Encode probability ...
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16058 cc @jkbradley @felixcheung --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14619: [SPARK-17031][SQL] Add `Scanner` operator to wrap...
Github user jiangxb1987 closed the pull request at: https://github.com/apache/spark/pull/14619 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15467: [SPARK-17912][SQL] Refactor code generation to get data ...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/15467 ping @ericl --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 I'm looking into the failures. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto p...
Github user uncleGen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16052#discussion_r89983105
--- Diff:
core/src/main/scala/org/apache/spark/serializer/SerializerManager.scala ---
@@ -155,7 +155,12 @@ private[spark] class
SerializerManager(defaultSerializer: Serializer, conf: Spar
outputStream: OutputStream,
values: Iterator[T]): Unit = {
val byteStream = new BufferedOutputStream(outputStream)
-val ser = getSerializer(implicitly[ClassTag[T]]).newInstance()
+val ser = blockId match {
--- End diff --
Ah, yours is better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16046: [SPARK-18582][SQL] Whitelist LogicalPlan operator...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16046#discussion_r89984727
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1011,24 +1011,24 @@ class Analyzer(
private def pullOutCorrelatedPredicates(sub: LogicalPlan):
(LogicalPlan, Seq[Expression]) = {
val predicateMap = scala.collection.mutable.Map.empty[LogicalPlan,
Seq[Expression]]
- /** Make sure a plans' subtree does not contain a tagged predicate.
*/
- def failOnOuterReferenceInSubTree(p: LogicalPlan, msg: String): Unit
= {
+ // Make sure a plan's subtree does not contain outer references
+ def failOnOuterReferenceInSubTree(p: LogicalPlan): Unit = {
if (p.collect(predicateMap).nonEmpty) {
--- End diff --
Lets change this line into `p.collectFirst(predicateMap).nonEmpty` that is
a little more efficient.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16046: [SPARK-18582][SQL] Whitelist LogicalPlan operator...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16046#discussion_r89975192
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1110,6 +1157,12 @@ class Analyzer(
} else {
p
}
+
+// Aggregate cannot host any correlated expressions
+// It can be on a correlation path if the correlation has
+// only equality correlated predicates.
+// It cannot be on a correlation path if the correlation has
--- End diff --
Nit: has -> contains?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16046: [SPARK-18582][SQL] Whitelist LogicalPlan operator...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16046#discussion_r89977826
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1120,47 +1173,54 @@ class Analyzer(
} else {
a
}
-case w : Window =>
- failOnOuterReference(w)
- failOnNonEqualCorrelatedPredicate(foundNonEqualCorrelatedPred, w)
- w
-case j @ Join(left, _, RightOuter, _) =>
- failOnOuterReference(j)
- failOnOuterReferenceInSubTree(left, "a RIGHT OUTER JOIN")
- j
-// SPARK-18578: Do not allow any correlated predicate
-// in a Full (Outer) Join operator and its descendants
-case j @ Join(_, _, FullOuter, _) =>
- failOnOuterReferenceInSubTree(j, "a FULL OUTER JOIN")
- j
-case j @ Join(_, right, jt, _) if !jt.isInstanceOf[InnerLike] =>
- failOnOuterReference(j)
- failOnOuterReferenceInSubTree(right, "a LEFT (OUTER) JOIN")
+
+// Join can host correlated expressions.
+case j @ Join(left, right, joinType, _) =>
+ joinType match {
+// Inner join, like Filter, can be anywhere.
+// LeftSemi is a special case of Inner join which returns
+// only the first matched row to the right table.
+case _: InnerLike | LeftSemi =>
--- End diff --
We cannot pull-out a predicate which is defined in the right side of a
`LeftSemi` join. `LeftSemi` join should be treated like all a `LeftOuter`
join.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16046: [SPARK-18582][SQL] Whitelist LogicalPlan operator...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16046#discussion_r89978030
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1120,47 +1173,54 @@ class Analyzer(
} else {
a
}
-case w : Window =>
- failOnOuterReference(w)
- failOnNonEqualCorrelatedPredicate(foundNonEqualCorrelatedPred, w)
- w
-case j @ Join(left, _, RightOuter, _) =>
- failOnOuterReference(j)
- failOnOuterReferenceInSubTree(left, "a RIGHT OUTER JOIN")
- j
-// SPARK-18578: Do not allow any correlated predicate
-// in a Full (Outer) Join operator and its descendants
-case j @ Join(_, _, FullOuter, _) =>
- failOnOuterReferenceInSubTree(j, "a FULL OUTER JOIN")
- j
-case j @ Join(_, right, jt, _) if !jt.isInstanceOf[InnerLike] =>
- failOnOuterReference(j)
- failOnOuterReferenceInSubTree(right, "a LEFT (OUTER) JOIN")
+
+// Join can host correlated expressions.
+case j @ Join(left, right, joinType, _) =>
+ joinType match {
+// Inner join, like Filter, can be anywhere.
+// LeftSemi is a special case of Inner join which returns
+// only the first matched row to the right table.
+case _: InnerLike | LeftSemi =>
+ failOnOuterReference(j)
+
+// Left outer join's right operand cannot be on a correlation
path.
+// LeftAnti and ExistenceJoin are special cases of LeftOuter.
+// Note that ExistenceJoin cannot be expressed externally in
both SQL and DataFrame
+// so it should not show up here in Analysis phase. This is
just a safety net.
+case LeftOuter | LeftAnti | ExistenceJoin(_) =>
+ failOnOuterReference(j)
+ failOnOuterReferenceInSubTree(right)
+
+// Likewise, Right outer join's left operand cannot be on a
correlation path.
+case RightOuter =>
+ failOnOuterReference(j)
+ failOnOuterReferenceInSubTree(left)
+
+// Any other join types not explicitly listed above,
+// including Full outer join, are treated as Category 4.
+case _ =>
+ failOnOuterReferenceInSubTree(j)
+ }
j
-case u: Union =>
- failOnOuterReferenceInSubTree(u, "a UNION")
- u
-case s: SetOperation =>
- failOnOuterReferenceInSubTree(s.right, "an INTERSECT/EXCEPT")
- s
-case e: Expand =>
- failOnOuterReferenceInSubTree(e, "an EXPAND")
- e
-case l : LocalLimit =>
- failOnOuterReferenceInSubTree(l, "a LIMIT")
- l
-// Since LIMIT is represented as GlobalLimit(, (LocalLimit
(, child))
-// and we are walking bottom up, we will fail on LocalLimit before
-// reaching GlobalLimit.
-// The code below is just a safety net.
-case g : GlobalLimit =>
- failOnOuterReferenceInSubTree(g, "a LIMIT")
- g
-case s : Sample =>
- failOnOuterReferenceInSubTree(s, "a TABLESAMPLE")
- s
+
+// Generator with join=true, i.e., expressed with
+// LATERAL VIEW [OUTER], similar to inner join,
+// allows to have correlation under it
+// but must not host any outer references.
+// Note:
+// Generator with join=false is treated as Category 4.
+case p @ Generate(generator, join, _, _, _, _) if (join) =>
--- End diff --
Pattern match directly on `join=true`, in this case: `case p @
Generate(generator, true, _, _, _, _) =>`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16046: [SPARK-18582][SQL] Whitelist LogicalPlan operator...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16046#discussion_r89974976
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1077,10 +1077,54 @@ class Analyzer(
// Simplify the predicates before pulling them out.
val transformed = BooleanSimplification(sub) transformUp {
-// WARNING:
-// Only Filter can host correlated expressions at this time
-// Anyone adding a new "case" below needs to add the call to
-// "failOnOuterReference" to disallow correlated expressions in it.
+
+// Whitelist operators allowed in a correlated subquery
+// There are 4 categories:
+// 1. Operators that are allowed anywhere in a correlated
subquery, and,
+//by definition of the operators, they cannot host outer
references.
+// 2. Operators that are allowed anywhere in a correlated subquery
+//so long as they do not host outer references.
+// 3. Operators that need special handlings. These operators are
+//Project, Filter, Join, Aggregate, and Generate.
+//
+// Any operators that are not in the above list are allowed
+// in a correlated subquery only if they are not on a correlation
path.
+// In other word, these operators are allowed only under a
correlation point.
+//
+// A correlation path is defined as the sub-tree of all the
operators that
+// are on the path from the operator hosting the correlated
expressions
+// up to the operator producing the correlated values.
+
+// Category 1:
+// Leaf node can be anywhere in a correlated subquery.
+case n: LeafNode =>
+ n
+// Category 2:
+// These operators can be anywhere in a correlated subquery.
+// so long as they do not host outer references in the operators.
+// SubqueryAlias can be anywhere in a correlated subquery.
+case p: SubqueryAlias =>
--- End diff --
You don't need to check `failOnOuterReference ` for `SubqueryAlias`,
`Distinct`, `Repartition` or `BroadcastHint`. These operators do not contain
expressions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16046: [SPARK-18582][SQL] Whitelist LogicalPlan operator...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16046#discussion_r89980037
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1120,47 +1173,54 @@ class Analyzer(
} else {
a
}
-case w : Window =>
- failOnOuterReference(w)
- failOnNonEqualCorrelatedPredicate(foundNonEqualCorrelatedPred, w)
- w
-case j @ Join(left, _, RightOuter, _) =>
- failOnOuterReference(j)
- failOnOuterReferenceInSubTree(left, "a RIGHT OUTER JOIN")
- j
-// SPARK-18578: Do not allow any correlated predicate
-// in a Full (Outer) Join operator and its descendants
-case j @ Join(_, _, FullOuter, _) =>
- failOnOuterReferenceInSubTree(j, "a FULL OUTER JOIN")
- j
-case j @ Join(_, right, jt, _) if !jt.isInstanceOf[InnerLike] =>
- failOnOuterReference(j)
- failOnOuterReferenceInSubTree(right, "a LEFT (OUTER) JOIN")
+
+// Join can host correlated expressions.
+case j @ Join(left, right, joinType, _) =>
+ joinType match {
+// Inner join, like Filter, can be anywhere.
+// LeftSemi is a special case of Inner join which returns
+// only the first matched row to the right table.
+case _: InnerLike | LeftSemi =>
+ failOnOuterReference(j)
+
+// Left outer join's right operand cannot be on a correlation
path.
+// LeftAnti and ExistenceJoin are special cases of LeftOuter.
+// Note that ExistenceJoin cannot be expressed externally in
both SQL and DataFrame
+// so it should not show up here in Analysis phase. This is
just a safety net.
+case LeftOuter | LeftAnti | ExistenceJoin(_) =>
+ failOnOuterReference(j)
+ failOnOuterReferenceInSubTree(right)
+
+// Likewise, Right outer join's left operand cannot be on a
correlation path.
+case RightOuter =>
+ failOnOuterReference(j)
+ failOnOuterReferenceInSubTree(left)
+
+// Any other join types not explicitly listed above,
+// including Full outer join, are treated as Category 4.
+case _ =>
+ failOnOuterReferenceInSubTree(j)
+ }
j
-case u: Union =>
- failOnOuterReferenceInSubTree(u, "a UNION")
- u
-case s: SetOperation =>
- failOnOuterReferenceInSubTree(s.right, "an INTERSECT/EXCEPT")
- s
-case e: Expand =>
- failOnOuterReferenceInSubTree(e, "an EXPAND")
- e
-case l : LocalLimit =>
- failOnOuterReferenceInSubTree(l, "a LIMIT")
- l
-// Since LIMIT is represented as GlobalLimit(, (LocalLimit
(, child))
-// and we are walking bottom up, we will fail on LocalLimit before
-// reaching GlobalLimit.
-// The code below is just a safety net.
-case g : GlobalLimit =>
- failOnOuterReferenceInSubTree(g, "a LIMIT")
- g
-case s : Sample =>
- failOnOuterReferenceInSubTree(s, "a TABLESAMPLE")
- s
+
+// Generator with join=true, i.e., expressed with
+// LATERAL VIEW [OUTER], similar to inner join,
+// allows to have correlation under it
+// but must not host any outer references.
+// Note:
+// Generator with join=false is treated as Category 4.
+case p @ Generate(generator, join, _, _, _, _) if (join) =>
--- End diff --
Is generate part of group 2 or group 3? I would say group 2?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16046: [SPARK-18582][SQL] Whitelist LogicalPlan operator...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/16046#discussion_r89979376
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1120,47 +1173,54 @@ class Analyzer(
} else {
a
}
-case w : Window =>
- failOnOuterReference(w)
- failOnNonEqualCorrelatedPredicate(foundNonEqualCorrelatedPred, w)
- w
-case j @ Join(left, _, RightOuter, _) =>
- failOnOuterReference(j)
- failOnOuterReferenceInSubTree(left, "a RIGHT OUTER JOIN")
- j
-// SPARK-18578: Do not allow any correlated predicate
-// in a Full (Outer) Join operator and its descendants
-case j @ Join(_, _, FullOuter, _) =>
- failOnOuterReferenceInSubTree(j, "a FULL OUTER JOIN")
- j
-case j @ Join(_, right, jt, _) if !jt.isInstanceOf[InnerLike] =>
- failOnOuterReference(j)
- failOnOuterReferenceInSubTree(right, "a LEFT (OUTER) JOIN")
+
+// Join can host correlated expressions.
+case j @ Join(left, right, joinType, _) =>
+ joinType match {
+// Inner join, like Filter, can be anywhere.
+// LeftSemi is a special case of Inner join which returns
+// only the first matched row to the right table.
+case _: InnerLike | LeftSemi =>
+ failOnOuterReference(j)
+
+// Left outer join's right operand cannot be on a correlation
path.
+// LeftAnti and ExistenceJoin are special cases of LeftOuter.
+// Note that ExistenceJoin cannot be expressed externally in
both SQL and DataFrame
+// so it should not show up here in Analysis phase. This is
just a safety net.
+case LeftOuter | LeftAnti | ExistenceJoin(_) =>
+ failOnOuterReference(j)
+ failOnOuterReferenceInSubTree(right)
+
+// Likewise, Right outer join's left operand cannot be on a
correlation path.
+case RightOuter =>
+ failOnOuterReference(j)
+ failOnOuterReferenceInSubTree(left)
+
+// Any other join types not explicitly listed above,
+// including Full outer join, are treated as Category 4.
+case _ =>
+ failOnOuterReferenceInSubTree(j)
+ }
j
-case u: Union =>
- failOnOuterReferenceInSubTree(u, "a UNION")
- u
-case s: SetOperation =>
- failOnOuterReferenceInSubTree(s.right, "an INTERSECT/EXCEPT")
- s
-case e: Expand =>
- failOnOuterReferenceInSubTree(e, "an EXPAND")
- e
-case l : LocalLimit =>
- failOnOuterReferenceInSubTree(l, "a LIMIT")
- l
-// Since LIMIT is represented as GlobalLimit(, (LocalLimit
(, child))
-// and we are walking bottom up, we will fail on LocalLimit before
-// reaching GlobalLimit.
-// The code below is just a safety net.
-case g : GlobalLimit =>
- failOnOuterReferenceInSubTree(g, "a LIMIT")
- g
-case s : Sample =>
- failOnOuterReferenceInSubTree(s, "a TABLESAMPLE")
- s
+
+// Generator with join=true, i.e., expressed with
+// LATERAL VIEW [OUTER], similar to inner join,
+// allows to have correlation under it
+// but must not host any outer references.
+// Note:
+// Generator with join=false is treated as Category 4.
+case p @ Generate(generator, join, _, _, _, _) if (join) =>
+ if (containsOuter(generator)) {
--- End diff --
I am not sure what is going on here. Why only check all the expressions in
the operator, when the generator contains an outer reference? Generate only has
one expression, the generator, so I think you can safely call
`failOnOuterReference(p)` directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15255: [SPARK-17680] [SQL] [TEST] Added a Testcase for Verifyin...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15255 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69318/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16028 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/15975#discussion_r89986253 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -76,9 +76,6 @@ class JDBCOptions( // the number of partitions val numPartitions = parameters.get(JDBC_NUM_PARTITIONS).map(_.toInt) --- End diff -- have we documented this behaviour? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16047: [SPARK-17783] [SQL] [BACKPORT-2.0] Hide Credentials in C...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/16047 LGTM - merging to 2.0. Thanks! Can you close? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16059: [SPARK-18625][ML] OneVsRestModel should support s...
GitHub user zhengruifeng opened a pull request: https://github.com/apache/spark/pull/16059 [SPARK-18625][ML] OneVsRestModel should support setFeaturesCol and setPredictionCol ## What changes were proposed in this pull request? add `setFeaturesCol` and `setPredictionCol` for `OneVsRestModel` ## How was this patch tested? added tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhengruifeng/spark ovrm_setCol Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16059.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16059 commit 30e6bf4acaba27808c437499cfa82659ac678c5c Author: Zheng RuiFeng Date: 2016-11-29T10:13:16Z create pr commit 47682bd6eb934e02ea53a80c3f364520b72b6d08 Author: Zheng RuiFeng Date: 2016-11-29T10:21:09Z update commit 4a6ecab1948943b1adbbf4f4418201c95901c628 Author: Zheng RuiFeng Date: 2016-11-29T11:07:30Z add test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16014 **[Test build #69315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69315/consoleFull)** for PR 16014 at commit [`c9c9802`](https://github.com/apache/spark/commit/c9c9802e67178e2283b1ebc9fa13f39db916773d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69315/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto pick" fe...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16052 **[Test build #69323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69323/consoleFull)** for PR 16052 at commit [`cd7e595`](https://github.com/apache/spark/commit/cd7e595338386211fe0e1da0e4e8605ff743d728). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16014 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16059: [SPARK-18625][ML] OneVsRestModel should support setFeatu...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16059 **[Test build #69322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69322/consoleFull)** for PR 16059 at commit [`4a6ecab`](https://github.com/apache/spark/commit/4a6ecab1948943b1adbbf4f4418201c95901c628). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16059: [SPARK-18625][ML] OneVsRestModel should support setFeatu...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16059 Is this not the same as https://github.com/apache/spark/pull/15957 but for a different class? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16048 **[Test build #69311 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69311/consoleFull)** for PR 16048 at commit [`fa313e5`](https://github.com/apache/spark/commit/fa313e5fdd60783df6fb96403802d4d1558b8cca). * This patch **fails executing the `dev/run-tests` script**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16048 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16048 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69311/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15620: [SPARK-18091] [SQL] Deep if expressions cause Generated ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15620 **[Test build #69324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69324/consoleFull)** for PR 15620 at commit [`6a7e9ac`](https://github.com/apache/spark/commit/6a7e9acde44155dc1e412da48bc14c0054ab973e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15620: [SPARK-18091] [SQL] Deep if expressions cause Generated ...
Github user kapilsingh5050 commented on the issue: https://github.com/apache/spark/pull/15620 The unit tests passed after incorporating related changes from SPARK-15327 and SPARK-17115 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15505: [SPARK-17931][CORE] taskScheduler has some unneeded seri...
Github user witgo commented on the issue: https://github.com/apache/spark/pull/15505 @kayousterhout Here are my thoughts: move the serialization out of the `TaskSetManager.resourceOffer` method. Split resourceOffer and serialization process, So that we can make the serialization task operation multi-threaded execution. See eb58988948341340432cf9cdde0d11842930ec57 . It can can reduce the scheduling time by three times --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
