[GitHub] [spark] yaooqinn commented on a change in pull request #27867: [SPARK-31111][SQL][Tests] Fix interval output issue in ExtractBenchmark
yaooqinn commented on a change in pull request #27867:
[SPARK-3][SQL][Tests] Fix interval output issue in ExtractBenchmark
URL: https://github.com/apache/spark/pull/27867#discussion_r390757478
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExtractBenchmark.scala
##
@@ -56,9 +56,11 @@ object ExtractBenchmark extends SqlBasedBenchmark {
}
}
- private def castExpr(from: String): String = from match {
+ private def castExpr(from: String, toStr: Boolean = false): String = from
match {
case "timestamp" => "cast(id as timestamp)"
case "date" => "cast(cast(id as timestamp) as date)"
+case "interval" if toStr => "cast((cast(cast(id as timestamp) as date) -
date'0001-01-01') + " +
Review comment:
good advice, thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on a change in pull request #27768: [SPARK-31018][CORE] Deprecate support of multiple workers on the same host in Standalone
jiangxb1987 commented on a change in pull request #27768: [SPARK-31018][CORE] Deprecate support of multiple workers on the same host in Standalone URL: https://github.com/apache/spark/pull/27768#discussion_r390756919 ## File path: docs/hardware-provisioning.md ## @@ -63,10 +63,10 @@ Note that memory usage is greatly affected by storage level and serialization fo the [tuning guide](tuning.html) for tips on how to reduce it. Finally, note that the Java VM does not always behave well with more than 200 GiB of RAM. If you -purchase machines with more RAM than this, you can run _multiple worker JVMs per node_. In -Spark's [standalone mode](spark-standalone.html), you can set the number of workers per node -with the `SPARK_WORKER_INSTANCES` variable in `conf/spark-env.sh`, and the number of cores -per worker with `SPARK_WORKER_CORES`. +purchase machines with more RAM than this, you can launch multiple executors in a single node. In +Spark's [standalone mode](spark-standalone.html), a worker is responsible for launching multiple +executors according to its available memory and cores. And each executor will be launched in a Review comment: nit: `And` -> `and` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on a change in pull request #27768: [SPARK-31018][CORE] Deprecate support of multiple workers on the same host in Standalone
jiangxb1987 commented on a change in pull request #27768: [SPARK-31018][CORE] Deprecate support of multiple workers on the same host in Standalone URL: https://github.com/apache/spark/pull/27768#discussion_r390756668 ## File path: docs/core-migration-guide.md ## @@ -38,3 +38,5 @@ license: | - Event log file will be written as UTF-8 encoding, and Spark History Server will replay event log files as UTF-8 encoding. Previously Spark writes event log file as default charset of driver JVM process, so Spark History Server of Spark 2.x is needed to read the old event log files in case of incompatible encoding. - A new protocol for fetching shuffle blocks is used. It's recommended that external shuffle services be upgraded when running Spark 3.0 apps. Old external shuffle services can still be used by setting the configuration `spark.shuffle.useOldFetchProtocol` to `true`. Otherwise, Spark may run into errors with messages like `IllegalArgumentException: Unexpected message type: `. + +- `SPARK_WORKER_INSTANCES` is deprecated. It's recommended to launch multiple executors in one worker and launch one worker per node instead of launching multiple workers per node and launching one executor per worker. Review comment: You should mention this is for Standalone backend only. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597453785 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119649/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597453780 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
SparkQA removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597449397 **[Test build #119649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119649/testReport)** for PR 27861 at commit [`85e66d2`](https://github.com/apache/spark/commit/85e66d27b5bfb53c88e01bd487c636225ba75ed2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597453758 **[Test build #119649 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119649/testReport)** for PR 27861 at commit [`85e66d2`](https://github.com/apache/spark/commit/85e66d27b5bfb53c88e01bd487c636225ba75ed2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597453785 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119649/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597453780 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27867: [SPARK-31111][SQL][Tests] Fix interval output issue in ExtractBenchmark
cloud-fan commented on a change in pull request #27867:
[SPARK-3][SQL][Tests] Fix interval output issue in ExtractBenchmark
URL: https://github.com/apache/spark/pull/27867#discussion_r390754252
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExtractBenchmark.scala
##
@@ -56,9 +56,11 @@ object ExtractBenchmark extends SqlBasedBenchmark {
}
}
- private def castExpr(from: String): String = from match {
+ private def castExpr(from: String, toStr: Boolean = false): String = from
match {
case "timestamp" => "cast(id as timestamp)"
case "date" => "cast(cast(id as timestamp) as date)"
+case "interval" if toStr => "cast((cast(cast(id as timestamp) as date) -
date'0001-01-01') + " +
Review comment:
Maybe we should not use the noop sink here, but
`df.queryExecution.toRDD.foreach(_ => ())`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597449635 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597449635 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597449644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24380/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597449644 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24380/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597449397 **[Test build #119649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119649/testReport)** for PR 27861 at commit [`85e66d2`](https://github.com/apache/spark/commit/85e66d27b5bfb53c88e01bd487c636225ba75ed2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27836: [SPARK-31009][SQL] Support json_object_keys function
AmplabJenkins removed a comment on issue #27836: [SPARK-31009][SQL] Support json_object_keys function URL: https://github.com/apache/spark/pull/27836#issuecomment-597448065 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24379/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597448066 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24378/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597448060 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27836: [SPARK-31009][SQL] Support json_object_keys function
AmplabJenkins removed a comment on issue #27836: [SPARK-31009][SQL] Support json_object_keys function URL: https://github.com/apache/spark/pull/27836#issuecomment-597448059 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597448060 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597448066 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24378/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27836: [SPARK-31009][SQL] Support json_object_keys function
AmplabJenkins commented on issue #27836: [SPARK-31009][SQL] Support json_object_keys function URL: https://github.com/apache/spark/pull/27836#issuecomment-597448065 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24379/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27836: [SPARK-31009][SQL] Support json_object_keys function
AmplabJenkins commented on issue #27836: [SPARK-31009][SQL] Support json_object_keys function URL: https://github.com/apache/spark/pull/27836#issuecomment-597448059 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597447725 **[Test build #119647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119647/testReport)** for PR 27861 at commit [`487e533`](https://github.com/apache/spark/commit/487e5333405b0c8c6d8140cecda5a608e78cb745). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27836: [SPARK-31009][SQL] Support json_object_keys function
SparkQA commented on issue #27836: [SPARK-31009][SQL] Support json_object_keys function URL: https://github.com/apache/spark/pull/27836#issuecomment-597447744 **[Test build #119648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119648/testReport)** for PR 27836 at commit [`7a74b30`](https://github.com/apache/spark/commit/7a74b303d209b67adfcdb3b902daed5ff8c3793e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem
AmplabJenkins commented on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem URL: https://github.com/apache/spark/pull/27801#issuecomment-597443657 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119639/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem
AmplabJenkins removed a comment on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem URL: https://github.com/apache/spark/pull/27801#issuecomment-597443657 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119639/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem
AmplabJenkins removed a comment on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem URL: https://github.com/apache/spark/pull/27801#issuecomment-597443653 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem
AmplabJenkins commented on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem URL: https://github.com/apache/spark/pull/27801#issuecomment-597443653 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem
SparkQA removed a comment on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem URL: https://github.com/apache/spark/pull/27801#issuecomment-597379727 **[Test build #119639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119639/testReport)** for PR 27801 at commit [`c6b964e`](https://github.com/apache/spark/commit/c6b964e0be490401a4d72e36b124f9029fd7e396). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem
SparkQA commented on issue #27801: [SPARK-31047][SQL] Improve file listing for ViewFileSystem URL: https://github.com/apache/spark/pull/27801#issuecomment-597443157 **[Test build #119639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119639/testReport)** for PR 27801 at commit [`c6b964e`](https://github.com/apache/spark/commit/c6b964e0be490401a4d72e36b124f9029fd7e396). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile closed pull request #27833: [SPARK-31070][SQL] make skew join split skewed partitions more evenly
gatorsmile closed pull request #27833: [SPARK-31070][SQL] make skew join split skewed partitions more evenly URL: https://github.com/apache/spark/pull/27833 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27833: [SPARK-31070][SQL] make skew join split skewed partitions more evenly
gatorsmile commented on issue #27833: [SPARK-31070][SQL] make skew join split skewed partitions more evenly URL: https://github.com/apache/spark/pull/27833#issuecomment-597443090 Thanks! Merged to master/3.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-597441984 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-597441984 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
AmplabJenkins commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-597441992 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119638/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
AmplabJenkins removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-597441992 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119638/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
AmplabJenkins commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597441544 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
AmplabJenkins commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597441551 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119646/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
SparkQA removed a comment on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-597375193 **[Test build #119638 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119638/testReport)** for PR 27803 at commit [`0ed5509`](https://github.com/apache/spark/commit/0ed5509b286521d05234a672a8634c77f3150557). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))
SparkQA commented on issue #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v)) URL: https://github.com/apache/spark/pull/27803#issuecomment-597441449 **[Test build #119638 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119638/testReport)** for PR 27803 at commit [`0ed5509`](https://github.com/apache/spark/commit/0ed5509b286521d05234a672a8634c77f3150557). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
SparkQA commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597441472 **[Test build #119646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119646/testReport)** for PR 27866 at commit [`d40afe2`](https://github.com/apache/spark/commit/d40afe2f10434144a993f4240e77c120cadd4ad2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
SparkQA removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597439142 **[Test build #119646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119646/testReport)** for PR 27866 at commit [`d40afe2`](https://github.com/apache/spark/commit/d40afe2f10434144a993f4240e77c120cadd4ad2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
AmplabJenkins removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597441551 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119646/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
AmplabJenkins removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597441544 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission
AmplabJenkins removed a comment on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission URL: https://github.com/apache/spark/pull/27832#issuecomment-597440947 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission
AmplabJenkins removed a comment on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission URL: https://github.com/apache/spark/pull/27832#issuecomment-597440951 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119642/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27867: [SPARK-31111][SQL][Tests] Fix interval output issue in ExtractBenchmark
yaooqinn commented on a change in pull request #27867:
[SPARK-3][SQL][Tests] Fix interval output issue in ExtractBenchmark
URL: https://github.com/apache/spark/pull/27867#discussion_r390743073
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExtractBenchmark.scala
##
@@ -56,9 +56,11 @@ object ExtractBenchmark extends SqlBasedBenchmark {
}
}
- private def castExpr(from: String): String = from match {
+ private def castExpr(from: String, toStr: Boolean = false): String = from
match {
case "timestamp" => "cast(id as timestamp)"
case "date" => "cast(cast(id as timestamp) as date)"
+case "interval" if toStr => "cast((cast(cast(id as timestamp) as date) -
date'0001-01-01') + " +
Review comment:
We forbid interval as output schema only
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission
AmplabJenkins commented on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission URL: https://github.com/apache/spark/pull/27832#issuecomment-597440951 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119642/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission
AmplabJenkins commented on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission URL: https://github.com/apache/spark/pull/27832#issuecomment-597440947 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow
cloud-fan commented on issue #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow URL: https://github.com/apache/spark/pull/27627#issuecomment-597440834 @skambha great analysis! I agree with you that we need another boolean flag in the sum aggregate buffer, but I'd like to make it simpler and only change it for decimals. How about we add a new expression `DecimalSum`? In which: 1. the buffer attributes are [sum, isEmpty] 2. initial value is [0, true] 3. the `updateExpression` should do: 3.1 update `isEmpty` to false 3.2 set `sum` to null if overflowed 3.3 do nothing if `sum` is already null. 4. the `mergeExpression` should do: 4.1 update `isEmpty` to false 4.2 if the input buffer's `isEmpty` is true, keep sum unchanged 4.3 if the input buffer's `isEmpty` is false, but `sum` is null, update its own `sum` to null 4.4 do nothing if `sum` is already null. 4.5 otherwise, add input buffer's `sum` 5. the `evaluateExpression` should do: 5.1 output null if `isEmpty` is true 5.2 fail if `sum` is null and ansi mode is on 5.3 otherwise, output the sum. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission
SparkQA removed a comment on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission URL: https://github.com/apache/spark/pull/27832#issuecomment-597400479 **[Test build #119642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119642/testReport)** for PR 27832 at commit [`7ffc615`](https://github.com/apache/spark/commit/7ffc615e4ae44b88709c47beb02ef5c8310cd64f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
viirya commented on a change in pull request #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#discussion_r390669221 ## File path: docs/sql-ref-syntax-qry-select-groupby.md ## @@ -44,7 +44,7 @@ GROUP BY [ GROUPING SETS grouping_sets ] group_expression [ , group_expression [ set is specified by a list of comma-separated expressions in parentheses. Syntax: -( () | ( expression [ , ... ] ) ) +( '()' | ( expression [ , ... ] ) ) Review comment: The syntax here indicates one group set or more? In above the syntax is `GROUP BY [ GROUPING SETS grouping_sets ] group_expression [ , group_expression [ , ... ] ]`. For example, `group_expression` here is for just one group expression. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission
SparkQA commented on issue #27832: [SPARK-31011][CORE] Log better message if SIGPWR is not supported while setting up decommission URL: https://github.com/apache/spark/pull/27832#issuecomment-597440486 **[Test build #119642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119642/testReport)** for PR 27832 at commit [`7ffc615`](https://github.com/apache/spark/commit/7ffc615e4ae44b88709c47beb02ef5c8310cd64f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
AmplabJenkins removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597439379 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24377/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
AmplabJenkins commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597439369 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
AmplabJenkins commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597439379 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24377/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
AmplabJenkins removed a comment on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597439369 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
SparkQA commented on issue #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT URL: https://github.com/apache/spark/pull/27866#issuecomment-597439142 **[Test build #119646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119646/testReport)** for PR 27866 at commit [`d40afe2`](https://github.com/apache/spark/commit/d40afe2f10434144a993f4240e77c120cadd4ad2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27866: [SPARK-31110][DOCS][SQL] refine sql doc for SELECT
cloud-fan commented on a change in pull request #27866:
[SPARK-31110][DOCS][SQL] refine sql doc for SELECT
URL: https://github.com/apache/spark/pull/27866#discussion_r390739939
##
File path: docs/sql-ref-syntax-qry-select.md
##
@@ -1,57 +1,61 @@
---
layout: global
title: SELECT
-displayTitle: SELECT
+displayTitle: SELECT
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
http://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
Spark supports a `SELECT` statement and conforms to the ANSI SQL standard.
Queries are
-used to retrieve result sets from one or more tables. The following section
+used to retrieve result sets from one or more tables. The following section
describes the overall query syntax and the sub-sections cover different
constructs
-of a query along with examples.
+of a query along with examples.
### Syntax
{% highlight sql %}
[ WITH with_query [ , ... ] ]
+select_statement [ { UNION | INTERSECT | EXCEPT } select_statement, ... ]
Review comment:
It's defined in `select_statement`:
https://github.com/apache/spark/pull/27866/files#diff-bc52347d9c1373190bce4389d3066ad5R40
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27867: [SPARK-31111][SQL][Tests] Fix interval output issue in ExtractBenchmark
cloud-fan commented on a change in pull request #27867:
[SPARK-3][SQL][Tests] Fix interval output issue in ExtractBenchmark
URL: https://github.com/apache/spark/pull/27867#discussion_r390739470
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExtractBenchmark.scala
##
@@ -56,9 +56,11 @@ object ExtractBenchmark extends SqlBasedBenchmark {
}
}
- private def castExpr(from: String): String = from match {
+ private def castExpr(from: String, toStr: Boolean = false): String = from
match {
case "timestamp" => "cast(id as timestamp)"
case "date" => "cast(cast(id as timestamp) as date)"
+case "interval" if toStr => "cast((cast(cast(id as timestamp) as date) -
date'0001-01-01') + " +
Review comment:
Why it's not a problem for date/timestamp?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange
AmplabJenkins removed a comment on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange URL: https://github.com/apache/spark/pull/27828#issuecomment-597435520 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange
AmplabJenkins commented on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange URL: https://github.com/apache/spark/pull/27828#issuecomment-597435524 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119636/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange
AmplabJenkins removed a comment on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange URL: https://github.com/apache/spark/pull/27828#issuecomment-597435524 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119636/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange
AmplabJenkins commented on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange URL: https://github.com/apache/spark/pull/27828#issuecomment-597435520 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange
SparkQA removed a comment on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange URL: https://github.com/apache/spark/pull/27828#issuecomment-597368045 **[Test build #119636 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119636/testReport)** for PR 27828 at commit [`1d9c8ff`](https://github.com/apache/spark/commit/1d9c8ff481852053210af8a425dae6bfb71f06b5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange
SparkQA commented on issue #27828: [SPARK-31068][SQL] Avoid IllegalArgumentException in broadcast exchange URL: https://github.com/apache/spark/pull/27828#issuecomment-597435003 **[Test build #119636 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119636/testReport)** for PR 27828 at commit [`1d9c8ff`](https://github.com/apache/spark/commit/1d9c8ff481852053210af8a425dae6bfb71f06b5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27807: [SPARK-31076][SQL] Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time
cloud-fan commented on a change in pull request #27807: [SPARK-31076][SQL]
Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time
URL: https://github.com/apache/spark/pull/27807#discussion_r390733273
##
File path:
sql/core/v1.2/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
##
@@ -136,7 +138,9 @@ public int getInt(int rowId) {
public long getLong(int rowId) {
int index = getRowIndex(rowId);
if (isTimestamp) {
- return timestampData.time[index] * 1000 + timestampData.nanos[index] /
1000 % 1000;
+ Timestamp ts = new Timestamp(timestampData.time[index]);
Review comment:
can we add a round trip test in `OrcQuerySuite` to read/write date before
1582?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597431026 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119641/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597431020 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
SparkQA removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597391069 **[Test build #119641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119641/testReport)** for PR 27861 at commit [`df63708`](https://github.com/apache/spark/commit/df63708f395c03748461792b472c5d6d17608b50). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597431026 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119641/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597431020 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-597430838 **[Test build #119641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119641/testReport)** for PR 27861 at commit [`df63708`](https://github.com/apache/spark/commit/df63708f395c03748461792b472c5d6d17608b50). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27867: [SPARK-31111][SQL][Tests] Fix interval output issue in ExtractBenchmark
yaooqinn commented on a change in pull request #27867:
[SPARK-3][SQL][Tests] Fix interval output issue in ExtractBenchmark
URL: https://github.com/apache/spark/pull/27867#discussion_r390732419
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExtractBenchmark.scala
##
@@ -56,9 +56,11 @@ object ExtractBenchmark extends SqlBasedBenchmark {
}
}
- private def castExpr(from: String): String = from match {
+ private def castExpr(from: String, toStr: Boolean = false): String = from
match {
case "timestamp" => "cast(id as timestamp)"
case "date" => "cast(cast(id as timestamp) as date)"
+case "interval" if toStr => "cast((cast(cast(id as timestamp) as date) -
date'0001-01-01') + " +
Review comment:
Hmm, in this benchmark, there are 2 kinds of tests, one is
`extract/date_part` and the other is `string to interval`, the latter fails
type checking when saving.
Or shall we just remove the `string to interval` case here
https://github.com/apache/spark/pull/27867/files#diff-b89131adf146fef7cd6e3db381fd0807L111
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27867: [SPARK-31111][SQL][Tests] Fix interval output issue in ExtractBenchmark
yaooqinn commented on a change in pull request #27867:
[SPARK-3][SQL][Tests] Fix interval output issue in ExtractBenchmark
URL: https://github.com/apache/spark/pull/27867#discussion_r390732419
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExtractBenchmark.scala
##
@@ -56,9 +56,11 @@ object ExtractBenchmark extends SqlBasedBenchmark {
}
}
- private def castExpr(from: String): String = from match {
+ private def castExpr(from: String, toStr: Boolean = false): String = from
match {
case "timestamp" => "cast(id as timestamp)"
case "date" => "cast(cast(id as timestamp) as date)"
+case "interval" if toStr => "cast((cast(cast(id as timestamp) as date) -
date'0001-01-01') + " +
Review comment:
Hmm, in this benchmark, there are 2 kinds of tests, one is
`extract/date_part` and the other is `string to interval`, the latter fails
type checking when saving out.
Or shall we just remove the `string to interval` case here
https://github.com/apache/spark/pull/27867/files#diff-b89131adf146fef7cd6e3db381fd0807L111
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27807: [SPARK-31076][SQL] Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time
cloud-fan commented on a change in pull request #27807: [SPARK-31076][SQL]
Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time
URL: https://github.com/apache/spark/pull/27807#discussion_r390732047
##
File path:
sql/core/v2.3/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
##
@@ -136,7 +138,9 @@ public int getInt(int rowId) {
public long getLong(int rowId) {
int index = getRowIndex(rowId);
if (isTimestamp) {
- return timestampData.time[index] * 1000 + timestampData.nanos[index] /
1000 % 1000;
+ Timestamp ts = new Timestamp(timestampData.time[index]);
+ ts.setNanos(timestampData.nanos[index]);
+ return DateTimeUtils.fromJavaTimestamp(ts);
Review comment:
Under the hood, this re-uses the `Timestamp` object and should be more
efficient.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27807: [SPARK-31076][SQL] Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time
cloud-fan commented on a change in pull request #27807: [SPARK-31076][SQL]
Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time
URL: https://github.com/apache/spark/pull/27807#discussion_r390731854
##
File path:
sql/core/v2.3/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
##
@@ -136,7 +138,9 @@ public int getInt(int rowId) {
public long getLong(int rowId) {
int index = getRowIndex(rowId);
if (isTimestamp) {
- return timestampData.time[index] * 1000 + timestampData.nanos[index] /
1000 % 1000;
+ Timestamp ts = new Timestamp(timestampData.time[index]);
+ ts.setNanos(timestampData.nanos[index]);
+ return DateTimeUtils.fromJavaTimestamp(ts);
Review comment:
We can use a simpler API:
`DateTimeUtils.fromJavaTimestamp(timestampData.asScratchTimestamp(index))`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
AmplabJenkins removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597429004 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119645/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
SparkQA removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597425408 **[Test build #119645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119645/testReport)** for PR 26319 at commit [`6495368`](https://github.com/apache/spark/commit/64953685796f0294a0c2cfb1e5a1bfcb800b1c46). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
AmplabJenkins removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597429000 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
AmplabJenkins commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597429000 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
SparkQA commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597428989 **[Test build #119645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119645/testReport)** for PR 26319 at commit [`6495368`](https://github.com/apache/spark/commit/64953685796f0294a0c2cfb1e5a1bfcb800b1c46). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
AmplabJenkins commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597429004 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119645/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final
AmplabJenkins commented on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final URL: https://github.com/apache/spark/pull/27870#issuecomment-597428641 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119640/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final
AmplabJenkins removed a comment on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final URL: https://github.com/apache/spark/pull/27870#issuecomment-597428635 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27867: [SPARK-31111][SQL][Tests] Fix interval output issue in ExtractBenchmark
cloud-fan commented on a change in pull request #27867:
[SPARK-3][SQL][Tests] Fix interval output issue in ExtractBenchmark
URL: https://github.com/apache/spark/pull/27867#discussion_r390730564
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ExtractBenchmark.scala
##
@@ -56,9 +56,11 @@ object ExtractBenchmark extends SqlBasedBenchmark {
}
}
- private def castExpr(from: String): String = from match {
+ private def castExpr(from: String, toStr: Boolean = false): String = from
match {
case "timestamp" => "cast(id as timestamp)"
case "date" => "cast(cast(id as timestamp) as date)"
+case "interval" if toStr => "cast((cast(cast(id as timestamp) as date) -
date'0001-01-01') + " +
Review comment:
why we need to cast to string?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final
AmplabJenkins commented on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final URL: https://github.com/apache/spark/pull/27870#issuecomment-597428635 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final
AmplabJenkins removed a comment on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final URL: https://github.com/apache/spark/pull/27870#issuecomment-597428641 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119640/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final
SparkQA removed a comment on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final URL: https://github.com/apache/spark/pull/27870#issuecomment-597391046 **[Test build #119640 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119640/testReport)** for PR 27870 at commit [`50b377d`](https://github.com/apache/spark/commit/50b377db1ea7e14bc5b0f094929f5b620710902c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final
SparkQA commented on issue #27870: [SPARK-31095][BUILD][2.4] Upgrade netty-all to 4.1.47.Final URL: https://github.com/apache/spark/pull/27870#issuecomment-597428338 **[Test build #119640 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119640/testReport)** for PR 27870 at commit [`50b377d`](https://github.com/apache/spark/commit/50b377db1ea7e14bc5b0f094929f5b620710902c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
AmplabJenkins commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597425701 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24376/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
AmplabJenkins removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597425701 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24376/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
AmplabJenkins removed a comment on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597425696 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
AmplabJenkins commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597425696 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
SparkQA commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597425408 **[Test build #119645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119645/testReport)** for PR 26319 at commit [`6495368`](https://github.com/apache/spark/commit/64953685796f0294a0c2cfb1e5a1bfcb800b1c46). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
HyukjinKwon commented on a change in pull request #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#discussion_r390726359 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala ## @@ -1901,6 +1909,7 @@ case class NestedStruct(f: ClassData) case class DeepNestedStruct(f: NestedStruct) case class InvalidInJava(`abstract`: Int) +private[sql] case class InvalidInJavaForNumber(`1something`: String) Review comment: I think you can just remove `private[sql] ` or use `private` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where…
HyukjinKwon commented on issue #26319: [SPARK-29594][SQL] Create a Dataset from a Sequence of Case class where… URL: https://github.com/apache/spark/pull/26319#issuecomment-597424122 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
AmplabJenkins removed a comment on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode URL: https://github.com/apache/spark/pull/27871#issuecomment-597423397 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode
AmplabJenkins commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode URL: https://github.com/apache/spark/pull/27871#issuecomment-597423687 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
