[GitHub] [spark] AmplabJenkins commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
AmplabJenkins commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560471137 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
AmplabJenkins commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560471145 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114729/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
SparkQA removed a comment on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560380115 **[Test build #114729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114729/testReport)** for PR 26195 at commit [`e270fea`](https://github.com/apache/spark/commit/e270fea3f9875614b81950c9630970cd4bc8aee4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
SparkQA commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560470120 **[Test build #114729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114729/testReport)** for PR 26195 at commit [`e270fea`](https://github.com/apache/spark/commit/e270fea3f9875614b81950c9630970cd4bc8aee4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26467: [SPARK-29477]Improve tooltip for Streaming tab
SparkQA removed a comment on issue #26467: [SPARK-29477]Improve tooltip for Streaming tab URL: https://github.com/apache/spark/pull/26467#issuecomment-560436371 **[Test build #4957 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4957/testReport)** for PR 26467 at commit [`dcfa29f`](https://github.com/apache/spark/commit/dcfa29ff8da42925a5703e588f2c382a3ab0cc78). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26467: [SPARK-29477]Improve tooltip for Streaming tab
SparkQA commented on issue #26467: [SPARK-29477]Improve tooltip for Streaming tab URL: https://github.com/apache/spark/pull/26467#issuecomment-560468479 **[Test build #4957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4957/testReport)** for PR 26467 at commit [`dcfa29f`](https://github.com/apache/spark/commit/dcfa29ff8da42925a5703e588f2c382a3ab0cc78). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26707: [SPARK-30076][BUILD][TESTS] Upgrade Mockito to 3.1.0
dongjoon-hyun commented on issue #26707: [SPARK-30076][BUILD][TESTS] Upgrade Mockito to 3.1.0 URL: https://github.com/apache/spark/pull/26707#issuecomment-560467661 +1, late LGTM~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amanomer commented on a change in pull request #26712: [SPARK-29883][SQL] Improve error messages when function name is an alias
amanomer commented on a change in pull request #26712: [SPARK-29883][SQL]
Improve error messages when function name is an alias
URL: https://github.com/apache/spark/pull/26712#discussion_r352688792
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/UnevaluableAggs.scala
##
@@ -53,7 +53,6 @@ abstract class UnevaluableBooleanAggBase(arg: Expression)
""",
since = "3.0.0")
case class BoolAnd(arg: Expression) extends UnevaluableBooleanAggBase(arg) {
Review comment:
Updated PR. cc @cloud-fan
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #26721: [SPARK-30085][SQL][DOC] Standardize sql reference
huaxingao commented on issue #26721: [SPARK-30085][SQL][DOC] Standardize sql reference URL: https://github.com/apache/spark/pull/26721#issuecomment-560465686 Thanks! @srowen @dilipbiswal This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26694: [MINOR][SQL] Rename config name to spark.sql.analyzer.failAmbiguousSelfJoin.enabled
dongjoon-hyun commented on issue #26694: [MINOR][SQL] Rename config name to spark.sql.analyzer.failAmbiguousSelfJoin.enabled URL: https://github.com/apache/spark/pull/26694#issuecomment-560464868 Thank you for update. +1, late LGTM! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26732: [MINOR][SS] Add implementation note on overriding serialize/deserialize in HDFSMetadataLog methods' scaladoc
dongjoon-hyun commented on issue #26732: [MINOR][SS] Add implementation note on overriding serialize/deserialize in HDFSMetadataLog methods' scaladoc URL: https://github.com/apache/spark/pull/26732#issuecomment-560464351 +1, Late LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder
dongjoon-hyun commented on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder URL: https://github.com/apache/spark/pull/26725#issuecomment-560462624 I'm back~ @HyukjinKwon . Is there any remaining issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] nchammas commented on issue #26719: [SPARK-30084] Document how to trigger Jekyll build on Python API doc changes
nchammas commented on issue #26719: [SPARK-30084] Document how to trigger Jekyll build on Python API doc changes URL: https://github.com/apache/spark/pull/26719#issuecomment-560462261 `safe` is [disabled by default](https://jekyllrb.com/docs/configuration/default/), and I cannot find anyplace in `docs/` where we've set it to something else. [My post on the Jekyll forum](https://talk.jekyllrb.com/t/should-jekyll-serve-watch-follow-symlinks/3659) didn't get any replies, unfortunately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26738: [SPARK-30082] Do not replace Zeros when replacing NaNs
AmplabJenkins removed a comment on issue #26738: [SPARK-30082] Do not replace Zeros when replacing NaNs URL: https://github.com/apache/spark/pull/26738#issuecomment-560455850 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26738: [SPARK-30082] Do not replace Zeros when replacing NaNs
AmplabJenkins commented on issue #26738: [SPARK-30082] Do not replace Zeros when replacing NaNs URL: https://github.com/apache/spark/pull/26738#issuecomment-560456369 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560455224 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114733/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26738: [SPARK-30082] Do not replace Zeros when replacing NaNs
AmplabJenkins commented on issue #26738: [SPARK-30082] Do not replace Zeros when replacing NaNs URL: https://github.com/apache/spark/pull/26738#issuecomment-560455850 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560455214 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560455224 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114733/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
SparkQA removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560405755 **[Test build #114733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114733/testReport)** for PR 26412 at commit [`0694e07`](https://github.com/apache/spark/commit/0694e0704f95be818dbc57eb060a8f3ea604e46d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] johnhany97 opened a new pull request #26738: [SPARK-30082] Do not replace Zeros when replacing NaNs
johnhany97 opened a new pull request #26738: [SPARK-30082] Do not replace Zeros
when replacing NaNs
URL: https://github.com/apache/spark/pull/26738
Fixes https://issues.apache.org/jira/browse/SPARK-30082
### What changes were proposed in this pull request?
Do not cast `NaN` to an `Integer`, `Long`, `Short` or `Byte`. This is
because casting `NaN` to those types results in a `0` which erroneously
replaces `0`s while only `NaN`s should be replaced.
### Why are the changes needed?
This Scala code snippet:
```
import scala.math;
println(Double.NaN.toLong)
```
returns `0` which is problematic as if you run the following Spark code,
`0`s get replaced as well:
```
>>> df = spark.createDataFrame([(1.0, 0), (0.0, 3), (float('nan'), 0)],
("index", "value"))
>>> df.show()
+-+-+
|index|value|
+-+-+
| 1.0|0|
| 0.0|3|
| NaN|0|
+-+-+
>>> df.replace(float('nan'), 2).show()
+-+-+
|index|value|
+-+-+
| 1.0|2|
| 0.0|3|
| 2.0|2|
+-+-+
```
### Does this PR introduce any user-facing change?
Yes, after the PR, running the same above code snippet returns the correct
expected results:
```
>>> df = spark.createDataFrame([(1.0, 0), (0.0, 3), (float('nan'), 0)],
("index", "value"))
>>> df.show()
+-+-+
|index|value|
+-+-+
| 1.0|0|
| 0.0|3|
| NaN|0|
+-+-+
>>> df.replace(float('nan'), 2).show()
+-+-+
|index|value|
+-+-+
| 1.0|0|
| 0.0|3|
| 2.0|0|
+-+-+
```
### How was this patch tested?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
SparkQA commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560454991 **[Test build #114733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114733/testReport)** for PR 26412 at commit [`0694e07`](https://github.com/apache/spark/commit/0694e0704f95be818dbc57eb060a8f3ea604e46d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560455214 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26737: [SPARK-30103][SQL] Consolidate Schema merge logic
AmplabJenkins removed a comment on issue #26737: [SPARK-30103][SQL] Consolidate Schema merge logic URL: https://github.com/apache/spark/pull/26737#issuecomment-560452440 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26737: [SPARK-30103][SQL] Consolidate Schema merge logic
AmplabJenkins commented on issue #26737: [SPARK-30103][SQL] Consolidate Schema merge logic URL: https://github.com/apache/spark/pull/26737#issuecomment-560453040 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26737: [SPARK-30103][SQL] Consolidate Schema merge logic
AmplabJenkins commented on issue #26737: [SPARK-30103][SQL] Consolidate Schema merge logic URL: https://github.com/apache/spark/pull/26737#issuecomment-560452440 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xy2953396112 commented on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax
xy2953396112 commented on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax URL: https://github.com/apache/spark/pull/26736#issuecomment-560452586 fix Conflicting files. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Fokko opened a new pull request #26737: [SPARK-30103][SQL] Consolidate Schema merge logic
Fokko opened a new pull request #26737: [SPARK-30103][SQL] Consolidate Schema merge logic URL: https://github.com/apache/spark/pull/26737 While working at https://github.com/apache/spark/pull/26644 I've noticed something strange in behavior. https://github.com/apache/spark/pull/26644 focusses on merging UserDefinedTypes into Spark's native types. Delta checks if the schema is still compatible, so as an integration test I tried to union two DF's, where one has a UserDefinedTypes, which should then be merged into a native type. To mimic this, I've used an union there, because we don't have the Delta extension, and Spark does not check schema compatibility on write, so it is impossible to reproduce the situation that we've observed with Delta. However, when Delta checks compatibility, it merges the schema using the `StructType.merge()`, and when Spark checks compatibility, it uses the `TypeCoercion`: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L314-L323 Since this is complex code, I think we should merge these two to get similar behavior. Curious what your opinion is on this, and why these so similar functions are kept separately. ### What changes were proposed in this pull request? Removing the `StructType.merge()`, and use `TypeCoercion.findTightestCommonType()` instead. The one in TypeCoercion looks more complete. ### Why are the changes needed? To simplify the codebase, and consolidate the behavior of merging schemas. ### Does this PR introduce any user-facing change? Not in UI/Console, possibly in behavior. ### How was this patch tested? Existing unit tests, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
nchammas commented on a change in pull request #24830: [SPARK-27990][SQL][ML]
Provide a way to recursively load data from datasource
URL: https://github.com/apache/spark/pull/24830#discussion_r352663360
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##
@@ -62,6 +62,10 @@ abstract class PartitioningAwareFileIndex(
pathGlobFilter.forall(_.accept(file.getPath))
}
+ protected lazy val recursiveFileLookup = {
+parameters.getOrElse("recursiveFileLookup", "false").toBoolean
Review comment:
FYI @Ngone51: #26718
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26641: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab.
SparkQA removed a comment on issue #26641: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab. URL: https://github.com/apache/spark/pull/26641#issuecomment-560434356 **[Test build #4956 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4956/testReport)** for PR 26641 at commit [`2df1f20`](https://github.com/apache/spark/commit/2df1f2093fc4780b258dc81d4f7c1108f709d7f2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26641: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab.
SparkQA commented on issue #26641: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab. URL: https://github.com/apache/spark/pull/26641#issuecomment-560447337 **[Test build #4956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4956/testReport)** for PR 26641 at commit [`2df1f20`](https://github.com/apache/spark/commit/2df1f2093fc4780b258dc81d4f7c1108f709d7f2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] XuQianJin-Stars commented on issue #26727: [SPARK-30087][CORE] Enhanced implementation of JmxSink on RMI remote calls
XuQianJin-Stars commented on issue #26727: [SPARK-30087][CORE] Enhanced implementation of JmxSink on RMI remote calls URL: https://github.com/apache/spark/pull/26727#issuecomment-560444342 hi @jerryshao Have time to review this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560443414 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114726/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
SparkQA removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560366258 **[Test build #114726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114726/testReport)** for PR 26412 at commit [`e89d806`](https://github.com/apache/spark/commit/e89d8068a55957a2d5930d0aa2bc3111b5010854). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560443403 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560443403 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560443414 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114726/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
SparkQA commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560443044 **[Test build #114726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114726/testReport)** for PR 26412 at commit [`e89d806`](https://github.com/apache/spark/commit/e89d8068a55957a2d5930d0aa2bc3111b5010854). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26076: [SPARK-29419][SQL] Fix Encoder thread-safety bug in createDataset(Seq)
AmplabJenkins commented on issue #26076: [SPARK-29419][SQL] Fix Encoder thread-safety bug in createDataset(Seq) URL: https://github.com/apache/spark/pull/26076#issuecomment-560439125 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19558/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26076: [SPARK-29419][SQL] Fix Encoder thread-safety bug in createDataset(Seq)
AmplabJenkins commented on issue #26076: [SPARK-29419][SQL] Fix Encoder thread-safety bug in createDataset(Seq) URL: https://github.com/apache/spark/pull/26076#issuecomment-560439113 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #26650: [SPARK-30067][CORE] Fix a bug in getBlockHosts
srowen commented on issue #26650: [SPARK-30067][CORE] Fix a bug in getBlockHosts URL: https://github.com/apache/spark/pull/26650#issuecomment-560438548 Eh, maybe CC @cloud-fan too for a look. I'm pretty confident this is a correct fix and worth a backport. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26467: [SPARK-29477]Improve tooltip for Streaming tab
SparkQA commented on issue #26467: [SPARK-29477]Improve tooltip for Streaming tab URL: https://github.com/apache/spark/pull/26467#issuecomment-560436371 **[Test build #4957 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4957/testReport)** for PR 26467 at commit [`dcfa29f`](https://github.com/apache/spark/commit/dcfa29ff8da42925a5703e588f2c382a3ab0cc78). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #26709: [SPARK-30075][CORE][TESTS] Fix the hashCode implementation of ArrayKeyIndexType correctly
srowen commented on issue #26709: [SPARK-30075][CORE][TESTS] Fix the hashCode implementation of ArrayKeyIndexType correctly URL: https://github.com/apache/spark/pull/26709#issuecomment-560435829 Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen closed pull request #26709: [SPARK-30075][CORE][TESTS] Fix the hashCode implementation of ArrayKeyIndexType correctly
srowen closed pull request #26709: [SPARK-30075][CORE][TESTS] Fix the hashCode implementation of ArrayKeyIndexType correctly URL: https://github.com/apache/spark/pull/26709 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen closed pull request #26721: [SPARK-30085][SQL][DOC] Standardize sql reference
srowen closed pull request #26721: [SPARK-30085][SQL][DOC] Standardize sql reference URL: https://github.com/apache/spark/pull/26721 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #26721: [SPARK-30085][SQL][DOC] Standardize sql reference
srowen commented on issue #26721: [SPARK-30085][SQL][DOC] Standardize sql reference URL: https://github.com/apache/spark/pull/26721#issuecomment-560435315 Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen closed pull request #26636: [SPARK-27721][BUILD] Switch to use right leveldbjni according to the platforms
srowen closed pull request #26636: [SPARK-27721][BUILD] Switch to use right leveldbjni according to the platforms URL: https://github.com/apache/spark/pull/26636 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #26636: [SPARK-27721][BUILD] Switch to use right leveldbjni according to the platforms
srowen commented on issue #26636: [SPARK-27721][BUILD] Switch to use right leveldbjni according to the platforms URL: https://github.com/apache/spark/pull/26636#issuecomment-560434652 Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26641: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab.
SparkQA commented on issue #26641: [SPARK-29453][WEBUI] Improve tooltips information for SQL tab. URL: https://github.com/apache/spark/pull/26641#issuecomment-560434356 **[Test build #4956 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4956/testReport)** for PR 26641 at commit [`2df1f20`](https://github.com/apache/spark/commit/2df1f2093fc4780b258dc81d4f7c1108f709d7f2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #26732: [MINOR][SS] Add implementation note on overriding serialize/deserialize in HDFSMetadataLog methods' scaladoc
srowen commented on issue #26732: [MINOR][SS] Add implementation note on overriding serialize/deserialize in HDFSMetadataLog methods' scaladoc URL: https://github.com/apache/spark/pull/26732#issuecomment-560433641 Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen closed pull request #26732: [MINOR][SS] Add implementation note on overriding serialize/deserialize in HDFSMetadataLog methods' scaladoc
srowen closed pull request #26732: [MINOR][SS] Add implementation note on overriding serialize/deserialize in HDFSMetadataLog methods' scaladoc URL: https://github.com/apache/spark/pull/26732 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen closed pull request #26693: [MINOR][SS] Note that writer should not be closed in RateStreamMicroBatchStream.serialize
srowen closed pull request #26693: [MINOR][SS] Note that writer should not be closed in RateStreamMicroBatchStream.serialize URL: https://github.com/apache/spark/pull/26693 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #26722: [SPARK-24666][ML] Fix infinity vectors produced by Word2Vec when numIterations are large
srowen commented on issue #26722: [SPARK-24666][ML] Fix infinity vectors produced by Word2Vec when numIterations are large URL: https://github.com/apache/spark/pull/26722#issuecomment-560431648 Ah right, disregard my previous comment. Am I right that the original implementation, being single-threaded, computes just one updated vector per word per iteration? and in the Spark implementation, it comes up with several, because the word may appear in multiple partitions. Then adding them doesn't make sense. It would make sense to average them. That's not quite the same as dividing by number of partitions, as the word may not appear in all partitions. You could accumulate a simple count in reduceByKey then divide through the sum by count? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder
AmplabJenkins removed a comment on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder URL: https://github.com/apache/spark/pull/26725#issuecomment-560429177 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder
AmplabJenkins removed a comment on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder URL: https://github.com/apache/spark/pull/26725#issuecomment-560429189 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114728/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder
AmplabJenkins commented on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder URL: https://github.com/apache/spark/pull/26725#issuecomment-560429177 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder
AmplabJenkins commented on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder URL: https://github.com/apache/spark/pull/26725#issuecomment-560429189 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114728/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder
SparkQA commented on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder URL: https://github.com/apache/spark/pull/26725#issuecomment-560428509 **[Test build #114728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114728/testReport)** for PR 26725 at commit [`62e2ef0`](https://github.com/apache/spark/commit/62e2ef019983b74c4b26fad240d21190fe890579). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder
SparkQA removed a comment on issue #26725: [DO-NOT-MERGE] Health Checking in PR Builder URL: https://github.com/apache/spark/pull/26725#issuecomment-560371870 **[Test build #114728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114728/testReport)** for PR 26725 at commit [`62e2ef0`](https://github.com/apache/spark/commit/62e2ef019983b74c4b26fad240d21190fe890579). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #26663: [SPARK-30025][Core] Continuous shuffle block fetching should be disabled by default when the old fetch protocol is used
xuanyuanking commented on issue #26663: [SPARK-30025][Core] Continuous shuffle block fetching should be disabled by default when the old fetch protocol is used URL: https://github.com/apache/spark/pull/26663#issuecomment-560421940 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location
tgravescs commented on issue #26633: [SPARK-29994][CORE] Add WILDCARD task location URL: https://github.com/apache/spark/pull/26633#issuecomment-560420362 In my opinion the ideal thing is to fix SPARK-18886, its the perfect time, this is a new major release and this isn't something impacting production now so we don't really need a "quick fix". I disagree with your comment there is no perfect solution, no one here has tried and really no one here has give me any metrics as to why this is so important of a fix. But I realize that is a lot more change so I'm ok with this going in as a temporary fix. Please update based on the comments made - I want to make sure this is clearly documented in the code has a hack that will go away and no on else should use it. Also can someone give me any performance metrics - how much of a different does the LocalShuffledRowRDD make? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax
AmplabJenkins removed a comment on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax URL: https://github.com/apache/spark/pull/26736#issuecomment-560413418 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114727/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax
AmplabJenkins removed a comment on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax URL: https://github.com/apache/spark/pull/26736#issuecomment-560413412 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax
AmplabJenkins commented on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax URL: https://github.com/apache/spark/pull/26736#issuecomment-560413412 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax
SparkQA removed a comment on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax URL: https://github.com/apache/spark/pull/26736#issuecomment-560371851 **[Test build #114727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114727/testReport)** for PR 26736 at commit [`2451d2f`](https://github.com/apache/spark/commit/2451d2f0932d6b26967af16f7fb54b46374c4e3e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax
AmplabJenkins commented on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax URL: https://github.com/apache/spark/pull/26736#issuecomment-560413418 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114727/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax
SparkQA commented on issue #26736: [SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax URL: https://github.com/apache/spark/pull/26736#issuecomment-560413053 **[Test build #114727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114727/testReport)** for PR 26736 at commit [`2451d2f`](https://github.com/apache/spark/commit/2451d2f0932d6b26967af16f7fb54b46374c4e3e). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560406300 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19557/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins removed a comment on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560406293 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560406293 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
AmplabJenkins commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560406300 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19557/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
SparkQA commented on issue #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#issuecomment-560405755 **[Test build #114733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114733/testReport)** for PR 26412 at commit [`0694e07`](https://github.com/apache/spark/commit/0694e0704f95be818dbc57eb060a8f3ea604e46d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
yaooqinn commented on a change in pull request #26412: [SPARK-29774][SQL] Date
and Timestamp type +/- null should be null as Postgres
URL: https://github.com/apache/spark/pull/26412#discussion_r352604626
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -246,6 +247,54 @@ class Analyzer(
CleanupAliases)
)
+ /**
+ * 1. Turns Add/Subtract of DateType/TimestampType/StringType and
CalendarIntervalType
+ *to TimeAdd/TimeSub.
+ * 2. Turns Add/Subtract of TimestampType/DateType/IntegerType
+ *and TimestampType/IntegerType/DateType to
DateAdd/DateSub/SubtractDates and
+ *to SubtractTimestamps.
+ * 3. Turns Multiply/Divide of CalendarIntervalType and NumericType
+ *to MultiplyInterval/DivideInterval
+ */
+ case class ResolveBinaryArithmetic(conf: SQLConf) extends Rule[LogicalPlan] {
+override def apply(plan: LogicalPlan): LogicalPlan =
plan.resolveOperatorsUp {
+ case p: LogicalPlan => p.transformExpressionsUp {
+case u @ UnresolvedAdd(l, r) if u.childrenResolved => (l.dataType,
r.dataType) match {
+ case (TimestampType | DateType | StringType, CalendarIntervalType) =>
+Cast(TimeAdd(l, r), l.dataType)
+ case (CalendarIntervalType, TimestampType | DateType | StringType) =>
+Cast(TimeAdd(r, l), r.dataType)
+ case (DateType, _) => DateAdd(l, r)
+ case (_, DateType) => DateAdd(r, l)
+ case (_, _) => Add(l, r)
+}
+case u @ UnresolvedSubtract(l, r) if u.childrenResolved =>
(l.dataType, r.dataType) match {
+ case (TimestampType | DateType | StringType, CalendarIntervalType) =>
+Cast(TimeSub(l, r), l.dataType)
+ case (CalendarIntervalType, TimestampType | DateType | StringType) =>
+Cast(TimeSub(r, l), r.dataType)
+ case (DateType | NullType, DateType) => if
(conf.usePostgreSQLDialect) {
+DateDiff(l, r)
+ } else {
+SubtractDates(l, r)
+ }
+ case (TimestampType, TimestampType | DateType | NullType) =>
SubtractTimestamps(l, r)
+ case (DateType | NullType, TimestampType) =>
SubtractTimestamps(Cast(l, TimestampType), r)
+ case (DateType, _) => DateSub(l, r)
+ case (_, _) => Subtract(l, r)
+}
+case u @ UnresolvedMultiply(l, r) if u.childrenResolved =>
(l.dataType, r.dataType) match {
+ case (CalendarIntervalType, _: NumericType | NullType) =>
MultiplyInterval(l, r)
+ case (_: NumericType | NullType, CalendarIntervalType) =>
MultiplyInterval(r, l)
Review comment:
`case (_, CalendarIntervalType) => MultiplyInterval(r, l)` might be enough
for this case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
yaooqinn commented on a change in pull request #26412: [SPARK-29774][SQL] Date
and Timestamp type +/- null should be null as Postgres
URL: https://github.com/apache/spark/pull/26412#discussion_r352602498
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -246,6 +247,54 @@ class Analyzer(
CleanupAliases)
)
+ /**
+ * 1. Turns Add/Subtract of DateType/TimestampType/StringType and
CalendarIntervalType
+ *to TimeAdd/TimeSub.
+ * 2. Turns Add/Subtract of TimestampType/DateType/IntegerType
+ *and TimestampType/IntegerType/DateType to
DateAdd/DateSub/SubtractDates and
+ *to SubtractTimestamps.
+ * 3. Turns Multiply/Divide of CalendarIntervalType and NumericType
+ *to MultiplyInterval/DivideInterval
+ */
+ case class ResolveBinaryArithmetic(conf: SQLConf) extends Rule[LogicalPlan] {
+override def apply(plan: LogicalPlan): LogicalPlan =
plan.resolveOperatorsUp {
+ case p: LogicalPlan => p.transformExpressionsUp {
+case u @ UnresolvedAdd(l, r) if u.childrenResolved => (l.dataType,
r.dataType) match {
+ case (TimestampType | DateType | StringType, CalendarIntervalType) =>
+Cast(TimeAdd(l, r), l.dataType)
+ case (CalendarIntervalType, TimestampType | DateType | StringType) =>
+Cast(TimeAdd(r, l), r.dataType)
+ case (DateType, _) => DateAdd(l, r)
+ case (_, DateType) => DateAdd(r, l)
+ case (_, _) => Add(l, r)
+}
+case u @ UnresolvedSubtract(l, r) if u.childrenResolved =>
(l.dataType, r.dataType) match {
+ case (TimestampType | DateType | StringType, CalendarIntervalType) =>
+Cast(TimeSub(l, r), l.dataType)
+ case (CalendarIntervalType, TimestampType | DateType | StringType) =>
+Cast(TimeSub(r, l), r.dataType)
+ case (DateType | NullType, DateType) => if
(conf.usePostgreSQLDialect) {
+DateDiff(l, r)
+ } else {
+SubtractDates(l, r)
+ }
+ case (TimestampType, TimestampType | DateType | NullType) =>
SubtractTimestamps(l, r)
+ case (DateType | NullType, TimestampType) =>
SubtractTimestamps(Cast(l, TimestampType), r)
+ case (DateType, _) => DateSub(l, r)
+ case (_, _) => Subtract(l, r)
+}
+case u @ UnresolvedMultiply(l, r) if u.childrenResolved =>
(l.dataType, r.dataType) match {
+ case (CalendarIntervalType, _: NumericType | NullType) =>
MultiplyInterval(l, r)
+ case (_: NumericType | NullType, CalendarIntervalType) =>
MultiplyInterval(r, l)
Review comment:
Here is another thing to discuss, do we have to only support `interval *
numeric` and`interval / numeric`, this is not same with other type coercion
rules, e.g. `1 / '2'` is valid, so `interval 1 day / '2'` should be valid too.
In PostgreSQL, also valid
```
postgres=# select interval '1' day * 2;
?column?
--
2 days
(1 row)
postgres=# select interval '1' day * '2';
?column?
--
2 days
(1 row)
postgres=# select interval '1' day / '2';
?column?
--
12:00:00
(1 row)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
AmplabJenkins removed a comment on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560400547 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19556/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
AmplabJenkins removed a comment on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560400541 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
AmplabJenkins commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560400541 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
AmplabJenkins commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560400547 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19556/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
yaooqinn commented on a change in pull request #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#discussion_r352598951 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -246,6 +247,54 @@ class Analyzer( CleanupAliases) ) + /** + * 1. Turns Add/Subtract of DateType/TimestampType/StringType and CalendarIntervalType Review comment: Update a bit ```scala /** * For [[UnresolvedAdd]]: * 1. If one side is timestamp/date/string and the other side is interval, turns it to * [[TimeAdd]]; * 2. else if one side is date, turns it to [[DateAdd]] ; * 3. else turns it to [[Add]]. * * For [[UnresolvedSubtract]]: * 1. If the left side is timestamp/date/string and the right side is an interval, turns it to * [[TimeSub]]; * 2. else if one side is timestamp and the other side is date/timestamp, turns it to * [[SubtractTimestamps]]; * 3. else if both side are dates, turns it to [[DateDiff]]/[[SubtractDates]]; * 4. else if the left side is date, turns it to [[DateSub]]; * 5. else turns it to [[Subtract]]. * * For [[UnresolvedMultiply]]: * 1. If one side is interval and the other side is numeric, turns it to [[MultiplyInterval]]; * 2. otherwise, turns it to [[Multiply]]. * * For [[UnresolvedDivide]]: * 1. If the left side is interval and the right side is numeric, turns it to * [[DivideInterval]]; * 2. otherwise, turns it to [[Divide]]. */ ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
SparkQA commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560399962 **[Test build #114732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114732/testReport)** for PR 26195 at commit [`e889cda`](https://github.com/apache/spark/commit/e889cda5b1ef4eb28dbdc276926a2318be3df531). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type
AmplabJenkins removed a comment on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type URL: https://github.com/apache/spark/pull/26577#issuecomment-560398950 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type
AmplabJenkins removed a comment on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type URL: https://github.com/apache/spark/pull/26577#issuecomment-560398958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114724/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type
AmplabJenkins commented on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type URL: https://github.com/apache/spark/pull/26577#issuecomment-560398950 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type
AmplabJenkins commented on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type URL: https://github.com/apache/spark/pull/26577#issuecomment-560398958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114724/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type
SparkQA removed a comment on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type URL: https://github.com/apache/spark/pull/26577#issuecomment-560347771 **[Test build #114724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114724/testReport)** for PR 26577 at commit [`ced5002`](https://github.com/apache/spark/commit/ced50026414c0c86cf263fa087ec00391347662c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
Ngone51 commented on a change in pull request #26195: [SPARK-29537][SQL] throw
exception when user defined a wrong base path
URL: https://github.com/apache/spark/pull/26195#discussion_r352596667
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##
@@ -221,7 +221,15 @@ abstract class PartitioningAwareFileIndex(
if (!fs.isDirectory(userDefinedBasePath)) {
throw new IllegalArgumentException(s"Option '$BASE_PATH_PARAM' must
be a directory")
}
-Set(fs.makeQualified(userDefinedBasePath))
+val qualifiedBasePath = fs.makeQualified(userDefinedBasePath)
+val qualifiedBasePathStr = qualifiedBasePath.toString
+rootPaths
+ .find(!fs.makeQualified(_).toString.startsWith(qualifiedBasePathStr))
Review comment:
Review note: I've inlined the `qualified()` function into `find()` clause.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type
SparkQA commented on issue #26577: [SPARK-29943][SQL] Improve error messages for unsupported data type URL: https://github.com/apache/spark/pull/26577#issuecomment-560398309 **[Test build #114724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114724/testReport)** for PR 26577 at commit [`ced5002`](https://github.com/apache/spark/commit/ced50026414c0c86cf263fa087ec00391347662c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 removed a comment on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
Ngone51 removed a comment on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560398175 Review note: I've inlined the `qualified()` function into `find` clause. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
Ngone51 commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-560398175 Review note: I've inlined the `qualified()` function into `find` clause. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path
Ngone51 commented on a change in pull request #26195: [SPARK-29537][SQL] throw
exception when user defined a wrong base path
URL: https://github.com/apache/spark/pull/26195#discussion_r352590490
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##
@@ -221,7 +221,16 @@ abstract class PartitioningAwareFileIndex(
if (!fs.isDirectory(userDefinedBasePath)) {
throw new IllegalArgumentException(s"Option '$BASE_PATH_PARAM' must
be a directory")
}
-Set(fs.makeQualified(userDefinedBasePath))
+def qualifiedPath(path: Path): String = fs.makeQualified(path).toString
+
+val qualifiedBasePath = qualifiedPath(userDefinedBasePath)
+rootPaths
+ .find(p => !qualifiedPath(p).startsWith(qualifiedBasePath))
+ .foreach { rp =>
+throw new IllegalArgumentException(
+ s"Wrong basePath $userDefinedBasePath for the root path: $rp")
+ }
+Set(new Path(qualifiedBasePath))
Review comment:
Ok, I see.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset
AmplabJenkins removed a comment on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset URL: https://github.com/apache/spark/pull/26681#issuecomment-560388815 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19555/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset
AmplabJenkins removed a comment on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset URL: https://github.com/apache/spark/pull/26681#issuecomment-560388805 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset
AmplabJenkins commented on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset URL: https://github.com/apache/spark/pull/26681#issuecomment-560388805 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset
AmplabJenkins commented on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset URL: https://github.com/apache/spark/pull/26681#issuecomment-560388815 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19555/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #26694: [MINOR][SQL] Rename config name to spark.sql.analyzer.failAmbiguousSelfJoin.enabled
cloud-fan closed pull request #26694: [MINOR][SQL] Rename config name to spark.sql.analyzer.failAmbiguousSelfJoin.enabled URL: https://github.com/apache/spark/pull/26694 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset
SparkQA commented on issue #26681: [SPARK-30048][SQL] Enable aggregates with interval type values for RelationalGroupedDataset URL: https://github.com/apache/spark/pull/26681#issuecomment-560388295 **[Test build #114731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114731/testReport)** for PR 26681 at commit [`6dca7b1`](https://github.com/apache/spark/commit/6dca7b1cb6d26cf347ce1d361474d5ec3067adf3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #26694: [MINOR][SQL] Rename config name to spark.sql.analyzer.failAmbiguousSelfJoin.enabled
cloud-fan commented on issue #26694: [MINOR][SQL] Rename config name to spark.sql.analyzer.failAmbiguousSelfJoin.enabled URL: https://github.com/apache/spark/pull/26694#issuecomment-560388042 merging to master! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres
cloud-fan commented on a change in pull request #26412: [SPARK-29774][SQL] Date and Timestamp type +/- null should be null as Postgres URL: https://github.com/apache/spark/pull/26412#discussion_r352583163 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -246,6 +247,54 @@ class Analyzer( CleanupAliases) ) + /** + * 1. Turns Add/Subtract of DateType/TimestampType/StringType and CalendarIntervalType Review comment: I think we need to carefully design the rule so that it's easy to reason about. How about ``` For `UnresolvedAdd`: 1. if one side is interval, turn it into `TimeAdd` 2. if one side is date, turn it into `DateAdd` 3. otherwise, turn it into `Add` For `UnresolvedSubtract`: 1. if the right side is interval, turn it into `TimeSub` 2. if one side is timestamp, turn in into `SubtractTimestamps` 3. if one side is date, turn it into `DateDiff`/`SubtractDates` 4. otherwise, turn it into `Subtract` ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected
AmplabJenkins removed a comment on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected URL: https://github.com/apache/spark/pull/26080#issuecomment-560385961 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19554/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected
AmplabJenkins removed a comment on issue #26080: [SPARK-29425][SQL] The ownership of a database should be respected URL: https://github.com/apache/spark/pull/26080#issuecomment-560385953 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
