[GitHub] [spark] AmplabJenkins removed a comment on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference
AmplabJenkins removed a comment on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference URL: https://github.com/apache/spark/pull/25522#issuecomment-523690848 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference
AmplabJenkins commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference URL: https://github.com/apache/spark/pull/25522#issuecomment-523690851 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109535/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference
SparkQA removed a comment on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference URL: https://github.com/apache/spark/pull/25522#issuecomment-523688729 **[Test build #109535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109535/testReport)** for PR 25522 at commit [`08ba5f6`](https://github.com/apache/spark/commit/08ba5f676a8f8bf26fbeefa146f322ca158f1410). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #25532: [SPARK-28790][DOC][SQL]Document CACHE TABLE statement in SQL Reference
huaxingao commented on issue #25532: [SPARK-28790][DOC][SQL]Document CACHE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25532#issuecomment-523690824  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference
SparkQA commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference URL: https://github.com/apache/spark/pull/25522#issuecomment-523690784 **[Test build #109535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109535/testReport)** for PR 25522 at commit [`08ba5f6`](https://github.com/apache/spark/commit/08ba5f676a8f8bf26fbeefa146f322ca158f1410). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
SparkQA removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-523625017 **[Test build #109523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109523/testReport)** for PR 25247 at commit [`5d9378a`](https://github.com/apache/spark/commit/5d9378ac42b2c0d43ae4e77b116b05e20496de21). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
SparkQA commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-523690647 **[Test build #109523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109523/testReport)** for PR 25247 at commit [`5d9378a`](https://github.com/apache/spark/commit/5d9378ac42b2c0d43ae4e77b116b05e20496de21). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class AppendDataExecV1(` * `case class OverwriteByExpressionExecV1(` * `sealed trait V1FallbackWriters extends SupportsV1Write ` * ` protected implicit class toV1WriteBuilder(builder: WriteBuilder) ` * `trait SupportsV1Write extends SparkPlan ` * `trait V1WriteBuilder extends WriteBuilder ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference
huaxingao commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25540#issuecomment-523693037  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference
SparkQA commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25540#issuecomment-523693196 **[Test build #109537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109537/testReport)** for PR 25540 at commit [`b3d2e6c`](https://github.com/apache/spark/commit/b3d2e6c328dd8a54b2676fe74a1c139a5684dd05). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #25536: [SPARK-28837][SQL] CTAS/RTAS should use nullable schema
maropu commented on issue #25536: [SPARK-28837][SQL] CTAS/RTAS should use nullable schema URL: https://github.com/apache/spark/pull/25536#issuecomment-523692881 +1; the change looks reasonable to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution
SparkQA commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution URL: https://github.com/apache/spark/pull/25456#issuecomment-523696708 **[Test build #109525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109525/testReport)** for PR 25456 at commit [`a40c8b9`](https://github.com/apache/spark/commit/a40c8b92c829817fd31a43a967f28f81d14c49ee). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL] Improve EXPLAIN command
dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL]
Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r316454665
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
##
@@ -682,6 +699,28 @@ abstract class BaseSubqueryExec extends SparkPlan {
override def outputPartitioning: Partitioning = child.outputPartitioning
override def outputOrdering: Seq[SortOrder] = child.outputOrdering
+
+ override def generateTreeString(
+ depth: Int,
+ lastChildren: Seq[Boolean],
+ append: String => Unit,
+ verbose: Boolean,
+ prefix: String = "",
+ addSuffix: Boolean = false,
+ maxFields: Int,
+ printNodeId: Boolean): Unit = {
+if (!printNodeId) {
Review comment:
@cloud-fan Oh.. because we don't show these SubqueryExec ops in the new
format. Here is an example:
** SQL **
```
explain select * from explain_temp1 where key = (select max(key) from
explain_temp2);
```
** Old output **
```
== Physical Plan ==
*(1) Project [key#33776, val#33777]
+- *(1) Filter (isnotnull(key#33776) AND (key#33776 = Subquery
scalar-subquery#33790))
: +- Subquery scalar-subquery#33790
: +- *(2) HashAggregate(keys=[], functions=[max(key#33778)])
:+- Exchange SinglePartition, true
: +- *(1) HashAggregate(keys=[],
functions=[partial_max(key#33778)])
: +- *(1) ColumnarToRow
: +- FileScan parquet default.explain_temp2[key#33778]
Batched: true, DataFilters: [], Format: Parquet, Location:
InMemoryFileIndex[file:/user/hive/warehouse/explain_temp2], PartitionFilters:
[], PushedFilters: [], ReadSchema: struct
+- *(1) ColumnarToRow
+- FileScan parquet default.explain_temp1[key#33776,val#33777]
Batched: true, DataFilters: [isnotnull(key#33776)], Format: Parquet, Location:
InMemoryFileIndex[file:/user/hive/warehouse/explain_temp1], PartitionFilters:
[], PushedFilters: [IsNotNull(key)], ReadSchema: struct
```
The line `Subquery scalar-subquery#33790` does not show up in the new plan
as we only show the actual plan and not the containing operator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25517: [SPARK-26895][CORE][branch-2.3] prepareSubmitEnvironment should be called within doAs for proxy users
SparkQA removed a comment on issue #25517: [SPARK-26895][CORE][branch-2.3] prepareSubmitEnvironment should be called within doAs for proxy users URL: https://github.com/apache/spark/pull/25517#issuecomment-523645178 **[Test build #109526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109526/testReport)** for PR 25517 at commit [`3249de3`](https://github.com/apache/spark/commit/3249de327f7372341dd66b7070123215329db561). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25517: [SPARK-26895][CORE][branch-2.3] prepareSubmitEnvironment should be called within doAs for proxy users
SparkQA commented on issue #25517: [SPARK-26895][CORE][branch-2.3] prepareSubmitEnvironment should be called within doAs for proxy users URL: https://github.com/apache/spark/pull/25517#issuecomment-523703074 **[Test build #109526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109526/testReport)** for PR 25517 at commit [`3249de3`](https://github.com/apache/spark/commit/3249de327f7372341dd66b7070123215329db561). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #25536: [SPARK-28837][SQL] CTAS/RTAS should use nullable schema
cloud-fan commented on a change in pull request #25536: [SPARK-28837][SQL]
CTAS/RTAS should use nullable schema
URL: https://github.com/apache/spark/pull/25536#discussion_r316461201
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
##
@@ -81,11 +81,12 @@ case class CreateTableAsSelectExec(
}
Utils.tryWithSafeFinallyAndFailureCallbacks({
+ val schema = query.schema.asNullable
Review comment:
I think it's too much work if we need to transform the logical plan and add
an extra Project to change the nullability.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #25536: [SPARK-28837][SQL] CTAS/RTAS should use nullable schema
maropu commented on a change in pull request #25536: [SPARK-28837][SQL]
CTAS/RTAS should use nullable schema
URL: https://github.com/apache/spark/pull/25536#discussion_r316463260
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
##
@@ -81,11 +81,12 @@ case class CreateTableAsSelectExec(
}
Utils.tryWithSafeFinallyAndFailureCallbacks({
+ val schema = query.schema.asNullable
Review comment:
Ah, I see. Thanks. Looks ok to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries
ulysses-you commented on issue #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries URL: https://github.com/apache/spark/pull/25464#issuecomment-523707693 Thanks for review this. Make hint consistent with repartition api is a good idea. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
AmplabJenkins commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523713914 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
AmplabJenkins removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523713914 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
AmplabJenkins removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523713916 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14595/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
AmplabJenkins commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523713916 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14595/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples
AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25412#issuecomment-523715442 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25453: [SPARK-28730][SQL] Configurable type coercion policy for table insertion
cloud-fan commented on issue #25453: [SPARK-28730][SQL] Configurable type coercion policy for table insertion URL: https://github.com/apache/spark/pull/25453#issuecomment-523715573 > Any thoughts on whether we can get rid of legacy mode for v1 if we implement ANSI mode? The general policy is to keep the legacy config for at least one release, if a behavior change is made. I think we can remove it in 3.1. I agree with forbidding legacy mode in v2. For now we can make v1 and v2 have different default policies. Once we make ANSI policy the default, then v1 and v2 can still have the same default policy. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples
AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25412#issuecomment-523715447 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14596/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25412#issuecomment-523715442 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25412#issuecomment-523715447 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14596/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true
SparkQA commented on issue #25520: [SPARK-28621][SQL] Make spark.sql.crossJoin.enabled default value true URL: https://github.com/apache/spark/pull/25520#issuecomment-523720404 **[Test build #109541 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109541/testReport)** for PR 25520 at commit [`3a0447f`](https://github.com/apache/spark/commit/3a0447ff052fb1df96155677ef291d2f8608d368). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage
SparkQA commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25508#issuecomment-523720415 **[Test build #109542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109542/testReport)** for PR 25508 at commit [`b7b7150`](https://github.com/apache/spark/commit/b7b715090c646a9aae81059948d85c6d1977e526). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
SparkQA commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523720417 **[Test build #109543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109543/testReport)** for PR 25500 at commit [`9bb03fd`](https://github.com/apache/spark/commit/9bb03fddb2dfd85f95214f9a71983f62f60ee84c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] keypointt commented on a change in pull request #25291: [SPARK-28554][SQL] implement basic catalog functionalities for JDBC v2 with a DS v1 fallback API
keypointt commented on a change in pull request #25291: [SPARK-28554][SQL] implement basic catalog functionalities for JDBC v2 with a DS v1 fallback API URL: https://github.com/apache/spark/pull/25291#discussion_r316406454 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala ## @@ -253,6 +260,16 @@ class FindDataSourceTable(sparkSession: SparkSession) extends Rule[LogicalPlan] case UnresolvedCatalogRelation(tableMeta) => DDLUtils.readHiveTable(tableMeta) + +case AppendData(DataSourceV2Relation(t: DataSourceV1Table, output, _), query, false) => + InsertIntoDataSourceCommand(fallbackToV1Relation(t, output), query, overwrite = false) + +case OverwriteByExpression( +DataSourceV2Relation(t: DataSourceV1Table, output, _), Literal(true, _), query, false) => Review comment: maybe also add a `@TODO` here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rdblue commented on a change in pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs
rdblue commented on a change in pull request #25465: [SPARK-28747][SQL] merge
the two data source v2 fallback configs
URL: https://github.com/apache/spark/pull/25465#discussion_r316411681
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/FileDataSourceV2FallBackSuite.scala
##
@@ -130,55 +130,9 @@ class FileDataSourceV2FallBackSuite extends QueryTest
with SharedSparkSession {
}
}
- test("Fall back to v1 when reading file with write only FileDataSourceV2") {
-val df = spark.range(10).toDF()
-withTempPath { file =>
- val path = file.getCanonicalPath
- // Dummy File writer should fail as expected.
Review comment:
Shouldn't this test always fall back to v1 as a result of the change? And
isn't that fallback what this was testing? I think this test should pass and
should still be valid.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
SparkQA removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-523556206 **[Test build #109506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109506/testReport)** for PR 25354 at commit [`6950650`](https://github.com/apache/spark/commit/6950650f67dcb68e6e4804bd116e65dba056d411). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
SparkQA commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-523656881 **[Test build #109506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109506/testReport)** for PR 25354 at commit [`6950650`](https://github.com/apache/spark/commit/6950650f67dcb68e6e4804bd116e65dba056d411). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25514: [SPARK-28784][SS]StreamExecution and StreamingQueryManager should utilize CheckpointFileManager to interact with checkpoint directories
SparkQA removed a comment on issue #25514: [SPARK-28784][SS]StreamExecution and StreamingQueryManager should utilize CheckpointFileManager to interact with checkpoint directories URL: https://github.com/apache/spark/pull/25514#issuecomment-523605271 **[Test build #109519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109519/testReport)** for PR 25514 at commit [`97bcf5b`](https://github.com/apache/spark/commit/97bcf5b810c4c312b9d8aedd90b0d8abd492c27c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25514: [SPARK-28784][SS]StreamExecution and StreamingQueryManager should utilize CheckpointFileManager to interact with checkpoint directories
SparkQA commented on issue #25514: [SPARK-28784][SS]StreamExecution and StreamingQueryManager should utilize CheckpointFileManager to interact with checkpoint directories URL: https://github.com/apache/spark/pull/25514#issuecomment-523667463 **[Test build #109519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109519/testReport)** for PR 25514 at commit [`97bcf5b`](https://github.com/apache/spark/commit/97bcf5b810c4c312b9d8aedd90b0d8abd492c27c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
imback82 commented on a change in pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#discussion_r316424686 ## File path: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala ## @@ -1909,7 +2020,6 @@ class DataSourceV2SQLSuite extends QueryTest with SharedSparkSession with Before } } - Review comment: Fixed. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
imback82 commented on a change in pull request #25247: [SPARK-28319][SQL]
Implement SHOW TABLES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25247#discussion_r316424445
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
##
@@ -42,6 +42,7 @@ object DataSourceV2Strategy extends Strategy with
PredicateHelper {
/**
* Pushes down filters to the data source reader
*
+ *
Review comment:
Fixed. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3
AmplabJenkins removed a comment on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3 URL: https://github.com/apache/spark/pull/25544#issuecomment-523681463 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14587/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3
AmplabJenkins removed a comment on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3 URL: https://github.com/apache/spark/pull/25544#issuecomment-523681457 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3
SparkQA commented on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3 URL: https://github.com/apache/spark/pull/25544#issuecomment-523681967 **[Test build #109532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109532/testReport)** for PR 25544 at commit [`fa3bdf6`](https://github.com/apache/spark/commit/fa3bdf6d16ca8b77943ed166733d33411ae80492). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25545: NETFLIX-BUILD: Set OMP_NUM_THREADS to executor cores for python.
SparkQA commented on issue #25545: NETFLIX-BUILD: Set OMP_NUM_THREADS to executor cores for python. URL: https://github.com/apache/spark/pull/25545#issuecomment-523681992 **[Test build #109531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109531/testReport)** for PR 25545 at commit [`7ffd6e4`](https://github.com/apache/spark/commit/7ffd6e4e51eaa8d909320bee4efe64c924802d0f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
SparkQA commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-523681986 **[Test build #109533 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109533/testReport)** for PR 24232 at commit [`a543c90`](https://github.com/apache/spark/commit/a543c900dc897522f5be43b598879459386f4144). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API
AmplabJenkins removed a comment on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-523684977 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109534/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference
SparkQA commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference URL: https://github.com/apache/spark/pull/25522#issuecomment-523688729 **[Test build #109535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109535/testReport)** for PR 25522 at commit [`08ba5f6`](https://github.com/apache/spark/commit/08ba5f676a8f8bf26fbeefa146f322ca158f1410). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference
AmplabJenkins removed a comment on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25540#issuecomment-523695454 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109537/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference
AmplabJenkins removed a comment on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25540#issuecomment-523695450 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference
SparkQA removed a comment on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25540#issuecomment-523693196 **[Test build #109537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109537/testReport)** for PR 25540 at commit [`b3d2e6c`](https://github.com/apache/spark/commit/b3d2e6c328dd8a54b2676fe74a1c139a5684dd05). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference
AmplabJenkins commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25540#issuecomment-523695450 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference
AmplabJenkins commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25540#issuecomment-523695454 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109537/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference
SparkQA commented on issue #25540: [SPARK-28830][DOC][SQL]Document UNCACHE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25540#issuecomment-523695392 **[Test build #109537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109537/testReport)** for PR 25540 at commit [`b3d2e6c`](https://github.com/apache/spark/commit/b3d2e6c328dd8a54b2676fe74a1c139a5684dd05). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3
SparkQA commented on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3 URL: https://github.com/apache/spark/pull/25544#issuecomment-523708762 **[Test build #109532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109532/testReport)** for PR 25544 at commit [`fa3bdf6`](https://github.com/apache/spark/commit/fa3bdf6d16ca8b77943ed166733d33411ae80492). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3
SparkQA removed a comment on issue #25544: [SPARK-17875][CORE][BUILD] Remove dependency on Netty 3 URL: https://github.com/apache/spark/pull/25544#issuecomment-523681967 **[Test build #109532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109532/testReport)** for PR 25544 at commit [`fa3bdf6`](https://github.com/apache/spark/commit/fa3bdf6d16ca8b77943ed166733d33411ae80492). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog
AmplabJenkins removed a comment on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog URL: https://github.com/apache/spark/pull/25507#issuecomment-523712431 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14594/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog
AmplabJenkins removed a comment on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog URL: https://github.com/apache/spark/pull/25507#issuecomment-523712425 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] akirillov commented on a change in pull request #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
akirillov commented on a change in pull request #25500: [SPARK-28778][MESOS]
Fixed executors advertised address when running in virtual network
URL: https://github.com/apache/spark/pull/25500#discussion_r316467338
##
File path:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
##
@@ -268,27 +268,19 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
.getOrElse {
throw new SparkException(s"Executor Spark home `$EXECUTOR_HOME` is
not set!")
}
- val runScript = new File(executorSparkHome, "./bin/spark-class").getPath
- command.setValue(
-"%s \"%s\" org.apache.spark.executor.CoarseGrainedExecutorBackend"
- .format(prefixEnv, runScript) +
-s" --driver-url $driverURL" +
-s" --executor-id $taskId" +
-s" --hostname ${executorHostname(offer)}" +
-s" --cores $numCores" +
-s" --app-id $appId")
+ val executable = new File(executorSparkHome, "./bin/spark-class").getPath
+ val runScript = "%s \"%s\"
org.apache.spark.executor.CoarseGrainedExecutorBackend"
Review comment:
There was a problem with interpolation - checkstyle was failing on the
following string:
```
val runScript = s"$prefixEnv \"$executable\"
org.apache.spark.executor.CoarseGrainedExecutorBackend"
```
Error:
```
error
file=/Users/akirillov/projects/mesosphere/spark/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
message=Expected token SEMI but got Token(STRING_LITERAL,"
org.apache.spark.executor.CoarseGrainedExecutorBackend",10094,"
org.apache.spark.executor.CoarseGrainedExecutorBackend")
```
The working variant with interpolation looks as following:
```
val runScript =
s"""$prefixEnv "$executable"
| org.apache.spark.executor.CoarseGrainedExecutorBackend
|""".stripMargin.replaceAll("\n", " ")
```
not sure if it's better than `String.format`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog
AmplabJenkins commented on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog URL: https://github.com/apache/spark/pull/25507#issuecomment-523712431 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14594/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog
AmplabJenkins commented on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog URL: https://github.com/apache/spark/pull/25507#issuecomment-523712425 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25412#issuecomment-523717007 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] triplesheep commented on issue #25538: Fix minor typo in SQLConf
triplesheep commented on issue #25538: Fix minor typo in SQLConf URL: https://github.com/apache/spark/pull/25538#issuecomment-523716945 @gengliangwang Done and the issue is SPARK-28844. Thanks for reviewing :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples
AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples URL: https://github.com/apache/spark/pull/25412#issuecomment-523717013 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14597/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] akirillov commented on a change in pull request #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
akirillov commented on a change in pull request #25500: [SPARK-28778][MESOS]
Fixed executors advertised address when running in virtual network
URL: https://github.com/apache/spark/pull/25500#discussion_r316473395
##
File path:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
##
@@ -268,27 +268,19 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
.getOrElse {
throw new SparkException(s"Executor Spark home `$EXECUTOR_HOME` is
not set!")
}
- val runScript = new File(executorSparkHome, "./bin/spark-class").getPath
- command.setValue(
-"%s \"%s\" org.apache.spark.executor.CoarseGrainedExecutorBackend"
- .format(prefixEnv, runScript) +
-s" --driver-url $driverURL" +
-s" --executor-id $taskId" +
-s" --hostname ${executorHostname(offer)}" +
-s" --cores $numCores" +
-s" --app-id $appId")
+ val executable = new File(executorSparkHome, "./bin/spark-class").getPath
+ val runScript = "%s \"%s\"
org.apache.spark.executor.CoarseGrainedExecutorBackend"
Review comment:
Ah, thanks for the pointer. Updated it alongside with too long test
description which failed checkstyle.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25509: [SPARK-28699][CORE][2.4] Fix a corner case for aborting indeterminate stage
AmplabJenkins removed a comment on issue #25509: [SPARK-28699][CORE][2.4] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25509#issuecomment-523546840 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109494/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25509: [SPARK-28699][CORE][2.4] Fix a corner case for aborting indeterminate stage
AmplabJenkins removed a comment on issue #25509: [SPARK-28699][CORE][2.4] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25509#issuecomment-523546832 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
huaxingao commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523556338  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
SparkQA commented on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-523556206 **[Test build #109506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109506/testReport)** for PR 25354 at commit [`6950650`](https://github.com/apache/spark/commit/6950650f67dcb68e6e4804bd116e65dba056d411). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
AmplabJenkins removed a comment on issue #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API URL: https://github.com/apache/spark/pull/25354#issuecomment-523555148 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14562/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
huaxingao commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523555927  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #25530: [SPARK-28805][DOC] Document DESCRIBE FUNCTION in SQL Reference
srowen commented on a change in pull request #25530: [SPARK-28805][DOC] Document DESCRIBE FUNCTION in SQL Reference URL: https://github.com/apache/spark/pull/25530#discussion_r316326559 ## File path: docs/sql-ref-syntax-aux-describe-function.md ## @@ -18,5 +18,40 @@ license: | See the License for the specific language governing permissions and limitations under the License. --- +### Description -**This page is under construction** +Return the metadata such as qualified function name, implementing class and usage of an existing function. +Input parameter function name may be optionally qualified. If the function is not qualified then the function is +resolved from the current database. If EXTENDED option is specified then additionally extended usage information Review comment: Back-tick `EXTENDED`, and add "the" before. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523556338  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive
dongjoon-hyun commented on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive URL: https://github.com/apache/spark/pull/25542#issuecomment-523587514 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hiv
AmplabJenkins removed a comment on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive URL: https://github.com/apache/spark/pull/25542#issuecomment-523496438 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25530: [SPARK-28805][DOC] Document DESCRIBE FUNCTION in SQL Reference
AmplabJenkins commented on issue #25530: [SPARK-28805][DOC] Document DESCRIBE FUNCTION in SQL Reference URL: https://github.com/apache/spark/pull/25530#issuecomment-523598173 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25530: [SPARK-28805][DOC] Document DESCRIBE FUNCTION in SQL Reference
AmplabJenkins commented on issue #25530: [SPARK-28805][DOC] Document DESCRIBE FUNCTION in SQL Reference URL: https://github.com/apache/spark/pull/25530#issuecomment-523598186 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14570/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25529: [SPARK-28804][DOC] Document DESCRIBE QUERY in SQL Reference
AmplabJenkins commented on issue #25529: [SPARK-28804][DOC] Document DESCRIBE QUERY in SQL Reference URL: https://github.com/apache/spark/pull/25529#issuecomment-523598162 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14571/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25529: [SPARK-28804][DOC] Document DESCRIBE QUERY in SQL Reference
AmplabJenkins commented on issue #25529: [SPARK-28804][DOC] Document DESCRIBE QUERY in SQL Reference URL: https://github.com/apache/spark/pull/25529#issuecomment-523598153 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive
AmplabJenkins commented on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive URL: https://github.com/apache/spark/pull/25542#issuecomment-523603395 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive
AmplabJenkins commented on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive URL: https://github.com/apache/spark/pull/25542#issuecomment-523603399 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109510/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive
SparkQA commented on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive URL: https://github.com/apache/spark/pull/25542#issuecomment-523603255 **[Test build #109510 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109510/testReport)** for PR 25542 at commit [`7e89f55`](https://github.com/apache/spark/commit/7e89f55edc66d144a776f3048a943a18275b5e49). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hiv
AmplabJenkins removed a comment on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive URL: https://github.com/apache/spark/pull/25542#issuecomment-523603399 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109510/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive
SparkQA removed a comment on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive URL: https://github.com/apache/spark/pull/25542#issuecomment-523592623 **[Test build #109510 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109510/testReport)** for PR 25542 at commit [`7e89f55`](https://github.com/apache/spark/commit/7e89f55edc66d144a776f3048a943a18275b5e49). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hiv
AmplabJenkins removed a comment on issue #25542: [SPARK-28840][SQL] conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive URL: https://github.com/apache/spark/pull/25542#issuecomment-523603395 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25529: [SPARK-28804][DOC] Document DESCRIBE QUERY in SQL Reference
SparkQA commented on issue #25529: [SPARK-28804][DOC] Document DESCRIBE QUERY in SQL Reference URL: https://github.com/apache/spark/pull/25529#issuecomment-523605701 **[Test build #109515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109515/testReport)** for PR 25529 at commit [`719c717`](https://github.com/apache/spark/commit/719c717d7c893c83d50f6d8374b0989bcdeedfda). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25539: [SPARK-28808][SQL] Document SHOW FUNCTIONS in SQL Reference.
SparkQA commented on issue #25539: [SPARK-28808][SQL] Document SHOW FUNCTIONS in SQL Reference. URL: https://github.com/apache/spark/pull/25539#issuecomment-523605320 **[Test build #109517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109517/testReport)** for PR 25539 at commit [`2088aa5`](https://github.com/apache/spark/commit/2088aa524185244719749fb9f77149bca4f40518). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25514: [SPARK-28784][SS]StreamExecution and StreamingQueryManager should utilize CheckpointFileManager to interact with checkpoint directories
SparkQA commented on issue #25514: [SPARK-28784][SS]StreamExecution and StreamingQueryManager should utilize CheckpointFileManager to interact with checkpoint directories URL: https://github.com/apache/spark/pull/25514#issuecomment-523605271 **[Test build #109519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109519/testReport)** for PR 25514 at commit [`97bcf5b`](https://github.com/apache/spark/commit/97bcf5b810c4c312b9d8aedd90b0d8abd492c27c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25531: [SPARK-28806][DOC] Document SHOW COLUMNS in SQL Reference
SparkQA commented on issue #25531: [SPARK-28806][DOC] Document SHOW COLUMNS in SQL Reference URL: https://github.com/apache/spark/pull/25531#issuecomment-523605250 **[Test build #109518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109518/testReport)** for PR 25531 at commit [`50ad094`](https://github.com/apache/spark/commit/50ad094c1737718ee4d3e76c941b38a1f9149af7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on a change in pull request #25536: [SPARK-28837][SQL] CTAS/RTAS should use nullable schema
brkyvz commented on a change in pull request #25536: [SPARK-28837][SQL]
CTAS/RTAS should use nullable schema
URL: https://github.com/apache/spark/pull/25536#discussion_r316357703
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
##
@@ -500,6 +499,32 @@ class DataSourceV2SQLSuite extends QueryTest with
SharedSparkSession with Before
assert(t.isInstanceOf[UnresolvedTable], "V1 table wasn't returned as an
unresolved table")
}
+ test("CreateTableAsSelect: nullable schema") {
+val basicCatalog = catalog("testcat").asTableCatalog
+val atomicCatalog = catalog("testcat_atomic").asTableCatalog
+val basicIdentifier = "testcat.table_name"
+val atomicIdentifier = "testcat_atomic.table_name"
+
+Seq((basicCatalog, basicIdentifier), (atomicCatalog,
atomicIdentifier)).foreach {
+ case (catalog, identifier) =>
+spark.sql(s"CREATE TABLE $identifier USING foo AS SELECT 1 i")
+
+val table = catalog.loadTable(Identifier.of(Array(), "table_name"))
Review comment:
It's better to me if you don't have to jump around too much when reading the
code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference
AmplabJenkins commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference URL: https://github.com/apache/spark/pull/25522#issuecomment-523612753 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference
AmplabJenkins commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference URL: https://github.com/apache/spark/pull/25522#issuecomment-523612764 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109520/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] brkyvz commented on issue #25521: [SPARK-28635][SQL][followup] CatalogManager should reflect the changes of default catalog
brkyvz commented on issue #25521: [SPARK-28635][SQL][followup] CatalogManager should reflect the changes of default catalog URL: https://github.com/apache/spark/pull/25521#issuecomment-523612457 Merging to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference
SparkQA commented on issue #25522: [SPARK-28787][DOC][SQL]Document LOAD DATA statement in SQL Reference URL: https://github.com/apache/spark/pull/25522#issuecomment-523612614 **[Test build #109520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109520/testReport)** for PR 25522 at commit [`19deb6d`](https://github.com/apache/spark/commit/19deb6dd55b7b289fdf324e4bd63c5ad0f656534). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #25524: [SPARK-28788][DOC][SQL]Document ANALYZE TABLE statement in SQL Reference
huaxingao commented on issue #25524: [SPARK-28788][DOC][SQL]Document ANALYZE TABLE statement in SQL Reference URL: https://github.com/apache/spark/pull/25524#issuecomment-523617420 Here is the screenshot  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-523627002 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-523627015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14580/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
AmplabJenkins removed a comment on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487#issuecomment-523637984 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
AmplabJenkins commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487#issuecomment-523638000 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109507/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
SparkQA removed a comment on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487#issuecomment-523571339 **[Test build #109507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109507/testReport)** for PR 25487 at commit [`1207e00`](https://github.com/apache/spark/commit/1207e00072e15c6ef905067d2e6f0a834c591129). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots
AmplabJenkins commented on issue #25487: [SPARK-28769][CORE] Improve warning message of BarrierExecutionMode when required slots > maximum slots URL: https://github.com/apache/spark/pull/25487#issuecomment-523637984 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow
SparkQA commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-523538354 **[Test build #109504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109504/testReport)** for PR 25461 at commit [`9b3fa4e`](https://github.com/apache/spark/commit/9b3fa4e1df8ee3b7a69fc38e84b24808dec4790f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rdblue commented on a change in pull request #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
rdblue commented on a change in pull request #25354: [SPARK-28612][SQL] Add
DataFrameWriterV2 API
URL: https://github.com/apache/spark/pull/25354#discussion_r316296208
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -3942,6 +3943,61 @@ object functions {
*/
def to_csv(e: Column): Column = to_csv(e, Map.empty[String, String].asJava)
+ /**
+ * A transform for timestamps and dates to partition data into years.
+ *
+ * @group partition_transforms
+ * @since 3.0.0
+ */
+ def years(e: Column): Column = withExpr { Years(e.expr) }
+
+ /**
+ * A transform for timestamps and dates to partition data into months.
+ *
+ * @group partition_transforms
+ * @since 3.0.0
+ */
+ def months(e: Column): Column = withExpr { Months(e.expr) }
+
+ /**
+ * A transform for timestamps and dates to partition data into days.
+ *
+ * @group partition_transforms
+ * @since 3.0.0
+ */
+ def days(e: Column): Column = withExpr { Days(e.expr) }
+
+ /**
+ * A transform for timestamps to partition data into hours.
+ *
+ * @group partition_transforms
+ * @since 3.0.0
+ */
+ def hours(e: Column): Column = withExpr { Hours(e.expr) }
+
+ /**
+ * A transform for any type that partitions by a hash of the input column.
+ *
+ * @group partition_transforms
+ * @since 3.0.0
+ */
+ def bucket(numBuckets: Column, e: Column): Column = withExpr {
+numBuckets.expr match {
+ case lit @ Literal(_, IntegerType) =>
+Bucket(lit, e.expr)
+ case _ =>
+throw new AnalysisException(s"Invalid number of buckets: $numBuckets")
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
srowen commented on a change in pull request #25500: [SPARK-28778][MESOS] Fixed
executors advertised address when running in virtual network
URL: https://github.com/apache/spark/pull/25500#discussion_r316302601
##
File path:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
##
@@ -268,27 +268,19 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
.getOrElse {
throw new SparkException(s"Executor Spark home `$EXECUTOR_HOME` is
not set!")
}
- val runScript = new File(executorSparkHome, "./bin/spark-class").getPath
- command.setValue(
-"%s \"%s\" org.apache.spark.executor.CoarseGrainedExecutorBackend"
- .format(prefixEnv, runScript) +
-s" --driver-url $driverURL" +
-s" --executor-id $taskId" +
-s" --hostname ${executorHostname(offer)}" +
-s" --cores $numCores" +
-s" --app-id $appId")
+ val executable = new File(executorSparkHome, "./bin/spark-class").getPath
+ val runScript = "%s \"%s\"
org.apache.spark.executor.CoarseGrainedExecutorBackend"
Review comment:
(The string interpolation was fine, but just leave it, no point changing
further)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
