[GitHub] [spark] SparkQA removed a comment on issue #28105: [SPARK-31316][SQL] SQLQueryTestSuite: Display the total generate time for generated java code.
SparkQA removed a comment on issue #28105: [SPARK-31316][SQL] SQLQueryTestSuite: Display the total generate time for generated java code. URL: https://github.com/apache/spark/pull/28105#issuecomment-608195146 **[Test build #120745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120745/testReport)** for PR 28105 at commit [`b61ff6f`](https://github.com/apache/spark/commit/b61ff6fe882c795a230fac05eb4a64803ed2fb14). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28105: [SPARK-31316][SQL] SQLQueryTestSuite: Display the total generate time for generated java code.
SparkQA commented on issue #28105: [SPARK-31316][SQL] SQLQueryTestSuite: Display the total generate time for generated java code. URL: https://github.com/apache/spark/pull/28105#issuecomment-608264915 **[Test build #120745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120745/testReport)** for PR 28105 at commit [`b61ff6f`](https://github.com/apache/spark/commit/b61ff6fe882c795a230fac05eb4a64803ed2fb14). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder
HyukjinKwon commented on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder URL: https://github.com/apache/spark/pull/27592#issuecomment-608263683 Let me tentatively backport this to branch-3.0. If we can't make it on time, I will revert it back from branch-3.0. Merged to branch-3.0 too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
Ngone51 commented on a change in pull request #28085:
[SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
URL: https://github.com/apache/spark/pull/28085#discussion_r402722262
##
File path:
core/src/main/scala/org/apache/spark/resource/TaskResourceRequests.scala
##
@@ -37,6 +38,8 @@ private[spark] class TaskResourceRequests() extends
Serializable {
def requests: Map[String, TaskResourceRequest] = _taskResources.asScala.toMap
+ def requestsJMap: JMap[String, TaskResourceRequest] =
_taskResources.asScala.toMap.asJava
Review comment:
ditto.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
Ngone51 commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r402722588 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1135,6 +1137,22 @@ private[spark] class DAGScheduler( } } + /** + * PythonRunner needs to know what the pyspark memory setting is for the profile being run. + * Pass it in the local properties of the task if it's set for the stage profile. Review comment: nit: indent. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
Ngone51 commented on a change in pull request #28085:
[SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
URL: https://github.com/apache/spark/pull/28085#discussion_r402769953
##
File path: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
##
@@ -106,6 +104,12 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
// Authentication helper used when serving method calls via socket from
Python side.
private lazy val authHelper = new SocketAuthHelper(conf)
+ // each python worker gets an equal part of the allocation. the worker pool
will grow to the
+ // number of concurrent tasks, which is determined by the number of cores in
this executor.
+ private def getWorkerMemoryMb(mem: Option[Long]): Option[Long] = {
+mem.map(_ / conf.get(EXECUTOR_CORES))
Review comment:
If so, `EXECUTOR_CORES` should also comes from `ResourceProfile`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
Ngone51 commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests URL: https://github.com/apache/spark/pull/28085#discussion_r402720901 ## File path: core/src/main/scala/org/apache/spark/api/java/JavaRDD.scala ## @@ -49,6 +50,20 @@ class JavaRDD[T](val rdd: RDD[T])(implicit val classTag: ClassTag[T]) */ def persist(newLevel: StorageLevel): JavaRDD[T] = wrapRDD(rdd.persist(newLevel)) + /** + * Specify a ResourceProfile to use when calculating this RDD. This is only supported on + * certain cluster managers and currently requires dynamic allocation to be enabled. + * It will result in new executors with the resources specified being acquired to + * calculate the RDD. + */ + def withResources(prof: ResourceProfile): JavaRDD[T] = wrapRDD(rdd.withResources(prof)) Review comment: Shall we use `rp` as parameter name to be consistent with Scala side? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
Ngone51 commented on a change in pull request #28085:
[SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
URL: https://github.com/apache/spark/pull/28085#discussion_r402722123
##
File path:
core/src/main/scala/org/apache/spark/resource/ExecutorResourceRequests.scala
##
@@ -38,6 +39,8 @@ private[spark] class ExecutorResourceRequests() extends
Serializable {
def requests: Map[String, ExecutorResourceRequest] =
_executorResources.asScala.toMap
+ def requestsJMap: JMap[String, ExecutorResourceRequest] =
_executorResources.asScala.toMap.asJava
Review comment:
Looks weirdCan't we just return `_executorResources`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28085: [SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
Ngone51 commented on a change in pull request #28085:
[SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests
URL: https://github.com/apache/spark/pull/28085#discussion_r402759339
##
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##
@@ -1135,6 +1137,22 @@ private[spark] class DAGScheduler(
}
}
+ /**
+ * PythonRunner needs to know what the pyspark memory setting is for the
profile being run.
+ * Pass it in the local properties of the task if it's set for the stage
profile.
+ */
+ private def addPysparkMemToProperties(stage: Stage, properties: Properties):
Unit = {
+val pysparkMem = if (stage.resourceProfileId ==
DEFAULT_RESOURCE_PROFILE_ID) {
+ logDebug("Using the default pyspark executor memory")
+ sc.conf.get(PYSPARK_EXECUTOR_MEMORY)
+} else {
+ val rp =
sc.resourceProfileManager.resourceProfileFromId(stage.resourceProfileId)
+ logDebug(s"Using profile ${stage.resourceProfileId} pyspark executor
memory")
+ rp.getPysparkMemory
+}
Review comment:
Can't default resource profile gets pySpark mem from `getPysparkMemory`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
AmplabJenkins removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-608262013 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120744/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
AmplabJenkins commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-608262013 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120744/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
AmplabJenkins commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-608262007 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
AmplabJenkins removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-608262007 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608261136 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120753/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
SparkQA commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-608261202 **[Test build #120744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120744/testReport)** for PR 27019 at commit [`d303b2f`](https://github.com/apache/spark/commit/d303b2fe2bc1ddc0b1e98c2028fb92322a0bb976). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait GeneratePredicateHelper extends PredicateHelper ` * `case class FilterExec(condition: Expression, child: SparkPlan)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
SparkQA removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608256014 **[Test build #120753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120753/testReport)** for PR 28104 at commit [`d777990`](https://github.com/apache/spark/commit/d7779908b3d1fe021592ed041921743b1ff9ca7f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608261131 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608261136 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120753/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
SparkQA removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-608189054 **[Test build #120744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120744/testReport)** for PR 27019 at commit [`d303b2f`](https://github.com/apache/spark/commit/d303b2fe2bc1ddc0b1e98c2028fb92322a0bb976). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608261131 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608260572 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120754/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
SparkQA removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608255991 **[Test build #120754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120754/testReport)** for PR 28087 at commit [`eade76e`](https://github.com/apache/spark/commit/eade76e8ccedf1792e0a5e947c91db09102dba41). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
SparkQA commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608261006 **[Test build #120753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120753/testReport)** for PR 28104 at commit [`d777990`](https://github.com/apache/spark/commit/d7779908b3d1fe021592ed041921743b1ff9ca7f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608260565 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608260565 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608260572 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120754/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
SparkQA commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608260422 **[Test build #120754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120754/testReport)** for PR 28087 at commit [`eade76e`](https://github.com/apache/spark/commit/eade76e8ccedf1792e0a5e947c91db09102dba41). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
cloud-fan commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-608259758 LGTM if tests pass This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer edited a comment on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder
beliefer edited a comment on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder URL: https://github.com/apache/spark/pull/27592#issuecomment-608218137 @HyukjinKwon You can merge this PR to branch-3.0 first. I have another worry about whether I manually port these PR back in branch-3.0 before release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
cloud-fan commented on a change in pull request #27207: [SPARK-18886][CORE]
Make Locality wait time measure resource under utilization due to delay
scheduling.
URL: https://github.com/apache/spark/pull/27207#discussion_r402765288
##
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##
@@ -543,6 +543,11 @@ package object config {
.version("1.2.0")
.fallbackConf(DYN_ALLOCATION_SCHEDULER_BACKLOG_TIMEOUT)
+ private[spark] val LEGACY_LOCALITY_WAIT_RESET =
+ConfigBuilder("spark.locality.wait.legacyResetOnTaskLaunch")
Review comment:
all the configs are end-user facing (they can set and it will take effect).
Internal config is for special use cases only (most of the users should not set
it).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
huaxingao commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608257531 @srowen Could you please check this one as well? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608256329 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25453/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608256354 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25452/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608256329 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25453/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
AmplabJenkins commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608256319 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608256346 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608256346 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608256354 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25452/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
AmplabJenkins removed a comment on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608256319 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
SparkQA commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608256014 **[Test build #120753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120753/testReport)** for PR 28104 at commit [`d777990`](https://github.com/apache/spark/commit/d7779908b3d1fe021592ed041921743b1ff9ca7f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference
SparkQA commented on issue #28087: [SPARK-31319][SQL][DOCS] Document UDFs/UDAFs in SQL Reference URL: https://github.com/apache/spark/pull/28087#issuecomment-608255991 **[Test build #120754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120754/testReport)** for PR 28087 at commit [`eade76e`](https://github.com/apache/spark/commit/eade76e8ccedf1792e0a5e947c91db09102dba41). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608244558 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25451/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608244558 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25451/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608244552 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
AmplabJenkins removed a comment on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608244552 Build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs
SparkQA commented on issue #28104: [SPARK-31331][SQL][DOCS] Document Spark integration with Hive UDFs/UDAFs/UDTFs URL: https://github.com/apache/spark/pull/28104#issuecomment-608244148 **[Test build #120752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120752/testReport)** for PR 28104 at commit [`67f9bf7`](https://github.com/apache/spark/commit/67f9bf7c75530dbbb62f280c59f4ec98b032d19e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #28099: [SPARK-31326][SQL][DOCS] Create Function docs structure for SQL Reference
huaxingao commented on issue #28099: [SPARK-31326][SQL][DOCS] Create Function docs structure for SQL Reference URL: https://github.com/apache/spark/pull/28099#issuecomment-608242337 Thanks you all! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-608239640 @tgravescs np. I believe I addressed all of you and @cloud-fan's comments and fixed the failing test (new failure after rebasing on latest code). Let me know if you have more comments or if you all come to a consensus on the legacy config having .internal() Do you have a preferred app for messaging? Pending LinkedIn connect request. I'd like to inquire about further testing, such as if there is a small cluster environment to test this on manually or any other suggested tips. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #28099: [SPARK-31326][SQL][DOCS] Create Function docs structure for SQL Reference
maropu commented on issue #28099: [SPARK-31326][SQL][DOCS] Create Function docs structure for SQL Reference URL: https://github.com/apache/spark/pull/28099#issuecomment-608239581 Thanks! Merged to master/3.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu closed pull request #28099: [SPARK-31326][SQL][DOCS] Create Function docs structure for SQL Reference
maropu closed pull request #28099: [SPARK-31326][SQL][DOCS] Create Function docs structure for SQL Reference URL: https://github.com/apache/spark/pull/28099 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-608238364 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25450/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-608238358 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-608238364 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25450/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-608238358 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
SparkQA commented on issue #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-608238148 **[Test build #120751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120751/testReport)** for PR 27207 at commit [`24c8ad9`](https://github.com/apache/spark/commit/24c8ad9ae23d360c7bb5bc813d04e6f5f6715d93). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608235721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120743/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
AmplabJenkins removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608235712 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r402742900
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -295,6 +295,41 @@ case class AlterViewAsCommand(
}
}
+/**
+ * A command for users to get views in the given database.
+ * If a databaseName is not given, the current database will be used.
+ * The syntax of using this command in SQL is:
+ * {{{
+ * SHOW VIEWS [(IN|FROM) database_name] [[LIKE] 'identifier_with_wildcards'];
+ * }}}
+ */
+case class ShowViewsCommand(
+databaseName: Option[String],
+tableIdentifierPattern: Option[String]) extends RunnableCommand {
+
+ // The result of SHOW VIEWS has three basic columns: namespace, viewName and
isTemporary.
+ override val output: Seq[Attribute] = Seq(
+AttributeReference("namespace", StringType, nullable = false)(),
+AttributeReference("viewName", StringType, nullable = false)(),
+AttributeReference("isTemporary", BooleanType, nullable = false)())
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+val catalog = sparkSession.sessionState.catalog
+val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+
+// Show the information of views.
+val views = tableIdentifierPattern.map(catalog.listViews(db, _))
+ .getOrElse(catalog.listViews(db, "*"))
+views.map { tableIdent =>
+ val namespace = tableIdent.database.getOrElse("")
Review comment:
Emm, I'll change the `toArray` back to `toSeq` when I get back to my well
configured dev machine...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
AmplabJenkins commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608235712 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
AmplabJenkins commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608235721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120743/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
SparkQA removed a comment on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608173496 **[Test build #120743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120743/testReport)** for PR 28040 at commit [`2c9ed55`](https://github.com/apache/spark/commit/2c9ed55045fd5ff685f651f1e9799fb94dff497a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric
SparkQA commented on issue #28040: [DO NOT MERGE][SPARK-31278][SS] Fix StreamingQuery output rows metric URL: https://github.com/apache/spark/pull/28040#issuecomment-608235152 **[Test build #120743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120743/testReport)** for PR 28040 at commit [`2c9ed55`](https://github.com/apache/spark/commit/2c9ed55045fd5ff685f651f1e9799fb94dff497a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make
Locality wait time measure resource under utilization due to delay scheduling.
URL: https://github.com/apache/spark/pull/27207#discussion_r402738705
##
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##
@@ -319,20 +336,38 @@ private[spark] class TaskSchedulerImpl(
taskSetsByStageIdAndAttempt -= manager.taskSet.stageId
}
}
+resetOnPreviousOffer -= manager.taskSet
manager.parent.removeSchedulable(manager)
logInfo(s"Removed TaskSet ${manager.taskSet.id}, whose tasks have all
completed, from pool" +
s" ${manager.parent.name}")
}
+ /**
+ * Offers resources to a single [[TaskSetManager]] at a given max allowed
[[TaskLocality]].
+ *
+ * @param taskSet task set to offer resources to
Review comment:
would you prefer I just change the doc, or the variable name as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#discussion_r402739821 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ## @@ -466,12 +503,28 @@ private[spark] class TaskSchedulerImpl( }.sum } + def minTaskLocality( Review comment: 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make
Locality wait time measure resource under utilization due to delay scheduling.
URL: https://github.com/apache/spark/pull/27207#discussion_r402739744
##
File path:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
##
@@ -196,6 +196,241 @@ class TaskSchedulerImplSuite extends SparkFunSuite with
LocalSparkContext with B
assert(!failedTaskSet)
}
+ def setupTaskScheduler(clock: ManualClock): TaskSchedulerImpl = {
+val conf = new SparkConf()
+sc = new SparkContext("local", "TaskSchedulerImplSuite", conf)
+val taskScheduler = new TaskSchedulerImpl(sc,
+ sc.conf.get(config.TASK_MAX_FAILURES),
+ clock = clock) {
+ override def createTaskSetManager(taskSet: TaskSet, maxTaskFailures:
Int): TaskSetManager = {
+new TaskSetManager(this, taskSet, maxTaskFailures,
blacklistTrackerOpt, clock)
+ }
+ override def shuffleOffers(offers: IndexedSeq[WorkerOffer]):
IndexedSeq[WorkerOffer] = {
+// Don't shuffle the offers around for this test. Instead, we'll just
pass in all
+// the permutations we care about directly.
+offers
+ }
+}
+// Need to initialize a DAGScheduler for the taskScheduler to use for
callbacks.
+new DAGScheduler(sc, taskScheduler) {
+ override def taskStarted(task: Task[_], taskInfo: TaskInfo): Unit = {}
+
+ override def executorAdded(execId: String, host: String): Unit = {}
+}
+taskScheduler.initialize(new FakeSchedulerBackend)
+val taskSet = FakeTask.createTaskSet(8, 1, 1,
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1"))
+)
+
+// Offer resources first so that when the taskset is submitted it can
initialize
+// with proper locality level. Otherwise, ANY would be the only locality
level.
+// See TaskSetManager.computeValidLocalityLevels()
+// This begins the task set as PROCESS_LOCAL locality level
+taskScheduler.resourceOffers(IndexedSeq(WorkerOffer("exec1", "host1", 1)))
+taskScheduler.submitTasks(taskSet)
+taskScheduler
+ }
+
+ test("SPARK-18886 - partial offers (isAllFreeResources = false) reset timer
before " +
+"any resources have been rejected") {
+val clock = new ManualClock()
+// All tasks created here are local to exec1, host1.
+// Locality level starts at PROCESS_LOCAL.
+val taskScheduler = setupTaskScheduler(clock)
+// Locality levels increase at 3000 ms.
+val advanceAmount = 3000
+
+// Advancing clock increases locality level to NODE_LOCAL.
+clock.advance(advanceAmount)
+
+// If there hasn't yet been any full resource offers,
+// partial resource (isAllFreeResources = false) offers reset delay
scheduling
+// if this and previous offers were accepted.
+// This line resets the timer and locality level is reset to PROCESS_LOCAL.
+assert(taskScheduler
+ .resourceOffers(
+IndexedSeq(WorkerOffer("exec1", "host1", 1)),
+isAllFreeResources = false)
+ .flatten.length === 1)
+
+// This NODE_LOCAL task should not be accepted.
+assert(taskScheduler
+ .resourceOffers(
+IndexedSeq(WorkerOffer("exec2", "host1", 1)),
+isAllFreeResources = false)
+ .flatten.isEmpty)
+ }
+
+ test("SPARK-18886 - delay scheduling timer is reset when it accepts all
resources offered when" +
+"isAllFreeResources = true") {
+val clock = new ManualClock()
+// All tasks created here are local to exec1, host1.
+// Locality level starts at PROCESS_LOCAL.
+val taskScheduler = setupTaskScheduler(clock)
+// Locality levels increase at 3000 ms.
+val advanceAmount = 3000
+
+// Advancing clock increases locality level to NODE_LOCAL.
+clock.advance(advanceAmount)
+
+// If there are no rejects on an all resource offer, delay scheduling is
reset.
+// This line resets the timer and locality level is reset to PROCESS_LOCAL.
+assert(taskScheduler
+ .resourceOffers(
+IndexedSeq(WorkerOffer("exec1", "host1", 1)),
+isAllFreeResources = true)
+ .flatten.length === 1)
+
+// This NODE_LOCAL task should not be accepted.
+assert(taskScheduler
+ .resourceOffers(
+IndexedSeq(WorkerOffer("exec2", "host1", 1)),
+isAllFreeResources = false)
+ .flatten.isEmpty)
+ }
+
+ test("SPARK-18886 - partial resource offers (isAllFreeResources = false)
reset " +
+"time if last full resource offer (isAllResources = true) was accepted as
well as any " +
+"following partial resource offers") {
+val clock = new ManualClock()
+// All tasks created here
[GitHub] [spark] bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make
Locality wait time measure resource under utilization due to delay scheduling.
URL: https://github.com/apache/spark/pull/27207#discussion_r402739751
##
File path:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
##
@@ -196,6 +196,241 @@ class TaskSchedulerImplSuite extends SparkFunSuite with
LocalSparkContext with B
assert(!failedTaskSet)
}
+ def setupTaskScheduler(clock: ManualClock): TaskSchedulerImpl = {
+val conf = new SparkConf()
+sc = new SparkContext("local", "TaskSchedulerImplSuite", conf)
+val taskScheduler = new TaskSchedulerImpl(sc,
+ sc.conf.get(config.TASK_MAX_FAILURES),
+ clock = clock) {
+ override def createTaskSetManager(taskSet: TaskSet, maxTaskFailures:
Int): TaskSetManager = {
+new TaskSetManager(this, taskSet, maxTaskFailures,
blacklistTrackerOpt, clock)
+ }
+ override def shuffleOffers(offers: IndexedSeq[WorkerOffer]):
IndexedSeq[WorkerOffer] = {
+// Don't shuffle the offers around for this test. Instead, we'll just
pass in all
+// the permutations we care about directly.
+offers
+ }
+}
+// Need to initialize a DAGScheduler for the taskScheduler to use for
callbacks.
+new DAGScheduler(sc, taskScheduler) {
+ override def taskStarted(task: Task[_], taskInfo: TaskInfo): Unit = {}
+
+ override def executorAdded(execId: String, host: String): Unit = {}
+}
+taskScheduler.initialize(new FakeSchedulerBackend)
+val taskSet = FakeTask.createTaskSet(8, 1, 1,
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1")),
+ Seq(TaskLocation("host1", "exec1"))
+)
+
+// Offer resources first so that when the taskset is submitted it can
initialize
+// with proper locality level. Otherwise, ANY would be the only locality
level.
+// See TaskSetManager.computeValidLocalityLevels()
+// This begins the task set as PROCESS_LOCAL locality level
+taskScheduler.resourceOffers(IndexedSeq(WorkerOffer("exec1", "host1", 1)))
+taskScheduler.submitTasks(taskSet)
+taskScheduler
+ }
+
+ test("SPARK-18886 - partial offers (isAllFreeResources = false) reset timer
before " +
+"any resources have been rejected") {
+val clock = new ManualClock()
+// All tasks created here are local to exec1, host1.
+// Locality level starts at PROCESS_LOCAL.
+val taskScheduler = setupTaskScheduler(clock)
+// Locality levels increase at 3000 ms.
+val advanceAmount = 3000
+
+// Advancing clock increases locality level to NODE_LOCAL.
+clock.advance(advanceAmount)
+
+// If there hasn't yet been any full resource offers,
+// partial resource (isAllFreeResources = false) offers reset delay
scheduling
+// if this and previous offers were accepted.
+// This line resets the timer and locality level is reset to PROCESS_LOCAL.
+assert(taskScheduler
+ .resourceOffers(
+IndexedSeq(WorkerOffer("exec1", "host1", 1)),
+isAllFreeResources = false)
+ .flatten.length === 1)
+
+// This NODE_LOCAL task should not be accepted.
+assert(taskScheduler
+ .resourceOffers(
+IndexedSeq(WorkerOffer("exec2", "host1", 1)),
+isAllFreeResources = false)
+ .flatten.isEmpty)
+ }
+
+ test("SPARK-18886 - delay scheduling timer is reset when it accepts all
resources offered when" +
+"isAllFreeResources = true") {
Review comment:
good 👀
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r402739332
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -295,6 +295,41 @@ case class AlterViewAsCommand(
}
}
+/**
+ * A command for users to get views in the given database.
+ * If a databaseName is not given, the current database will be used.
+ * The syntax of using this command in SQL is:
+ * {{{
+ * SHOW VIEWS [(IN|FROM) database_name] [[LIKE] 'identifier_with_wildcards'];
+ * }}}
+ */
+case class ShowViewsCommand(
+databaseName: Option[String],
+tableIdentifierPattern: Option[String]) extends RunnableCommand {
+
+ // The result of SHOW VIEWS has three basic columns: namespace, viewName and
isTemporary.
+ override val output: Seq[Attribute] = Seq(
+AttributeReference("namespace", StringType, nullable = false)(),
+AttributeReference("viewName", StringType, nullable = false)(),
+AttributeReference("isTemporary", BooleanType, nullable = false)())
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+val catalog = sparkSession.sessionState.catalog
+val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+
+// Show the information of views.
+val views = tableIdentifierPattern.map(catalog.listViews(db, _))
+ .getOrElse(catalog.listViews(db, "*"))
+views.map { tableIdent =>
+ val namespace = tableIdent.database.getOrElse("")
Review comment:
Find it back in 7bd9cc662528a7fcfdb1f07b6565a0e6cac8512a. Sorry for the
mistake.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r402739163
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -295,6 +295,41 @@ case class AlterViewAsCommand(
}
}
+/**
+ * A command for users to get views in the given database.
+ * If a databaseName is not given, the current database will be used.
+ * The syntax of using this command in SQL is:
+ * {{{
+ * SHOW VIEWS [(IN|FROM) database_name] [[LIKE] 'identifier_with_wildcards'];
+ * }}}
+ */
+case class ShowViewsCommand(
+databaseName: Option[String],
Review comment:
Yea, the database name should have been resolved. Thanks! Updated in
7bd9cc662528a7fcfdb1f07b6565a0e6cac8512a.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on a change in pull request #27207: [SPARK-18886][CORE] Make
Locality wait time measure resource under utilization due to delay scheduling.
URL: https://github.com/apache/spark/pull/27207#discussion_r402738705
##
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##
@@ -319,20 +336,38 @@ private[spark] class TaskSchedulerImpl(
taskSetsByStageIdAndAttempt -= manager.taskSet.stageId
}
}
+resetOnPreviousOffer -= manager.taskSet
manager.parent.removeSchedulable(manager)
logInfo(s"Removed TaskSet ${manager.taskSet.id}, whose tasks have all
completed, from pool" +
s" ${manager.parent.name}")
}
+ /**
+ * Offers resources to a single [[TaskSetManager]] at a given max allowed
[[TaskLocality]].
+ *
+ * @param taskSet task set to offer resources to
Review comment:
would you prefer I just change the doc, or the variable name as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time
cloud-fan commented on issue #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time URL: https://github.com/apache/spark/pull/28101#issuecomment-608224062 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time
cloud-fan closed pull request #28101: [SPARK-31328][SQL] Fix rebasing of overlapped local timestamps during daylight saving time URL: https://github.com/apache/spark/pull/28101 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-608221321 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25449/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
AmplabJenkins removed a comment on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-608221319 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-608221321 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25449/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
AmplabJenkins commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-608221319 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
SparkQA commented on issue #27897: [SPARK-31113][SQL] Add SHOW VIEWS command URL: https://github.com/apache/spark/pull/27897#issuecomment-608220952 **[Test build #120750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120750/testReport)** for PR 27897 at commit [`7bd9cc6`](https://github.com/apache/spark/commit/7bd9cc662528a7fcfdb1f07b6565a0e6cac8512a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder
beliefer commented on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder URL: https://github.com/apache/spark/pull/27592#issuecomment-608218137 @HyukjinKwon You can merge this PR to branch-3.0 first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf
gengliangwang closed pull request #28097: [SPARK-31325][SQL][Web UI] Control a plan explain mode in the events of SQL listeners via SQLConf URL: https://github.com/apache/spark/pull/28097 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP)
AmplabJenkins commented on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP) URL: https://github.com/apache/spark/pull/28026#issuecomment-608217794 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120742/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP)
AmplabJenkins removed a comment on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP) URL: https://github.com/apache/spark/pull/28026#issuecomment-608217794 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120742/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP)
AmplabJenkins removed a comment on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP) URL: https://github.com/apache/spark/pull/28026#issuecomment-608217792 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP)
AmplabJenkins commented on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP) URL: https://github.com/apache/spark/pull/28026#issuecomment-608217792 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP)
SparkQA removed a comment on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP) URL: https://github.com/apache/spark/pull/28026#issuecomment-608151542 **[Test build #120742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120742/testReport)** for PR 28026 at commit [`7204dd4`](https://github.com/apache/spark/commit/7204dd40f1e0de1fa99241e538b62dfcc7fd3058). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP)
SparkQA commented on issue #28026: [SPARK-31257][SQL] Unify create table syntax (WIP) URL: https://github.com/apache/spark/pull/28026#issuecomment-608217400 **[Test build #120742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120742/testReport)** for PR 28026 at commit [`7204dd4`](https://github.com/apache/spark/commit/7204dd40f1e0de1fa99241e538b62dfcc7fd3058). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SerdeInfo(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-608216642 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120749/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-608216636 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
SparkQA commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-608216615 **[Test build #120749 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120749/testReport)** for PR 28102 at commit [`0fe7bfd`](https://github.com/apache/spark/commit/0fe7bfd7d1109dc2d77f2ec15663670ea22fa29f). * This patch **fails Java style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-608216636 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
SparkQA removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-608214610 **[Test build #120749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120749/testReport)** for PR 28102 at commit [`0fe7bfd`](https://github.com/apache/spark/commit/0fe7bfd7d1109dc2d77f2ec15663670ea22fa29f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-608216642 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120749/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate
AmplabJenkins commented on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate URL: https://github.com/apache/spark/pull/28089#issuecomment-608215581 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120741/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate
AmplabJenkins commented on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate URL: https://github.com/apache/spark/pull/28089#issuecomment-608215577 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate
AmplabJenkins removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate URL: https://github.com/apache/spark/pull/28089#issuecomment-608215581 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120741/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate
AmplabJenkins removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate URL: https://github.com/apache/spark/pull/28089#issuecomment-608215577 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
Eric5553 commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r402723336
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -295,6 +295,41 @@ case class AlterViewAsCommand(
}
}
+/**
+ * A command for users to get views in the given database.
+ * If a databaseName is not given, the current database will be used.
+ * The syntax of using this command in SQL is:
+ * {{{
+ * SHOW VIEWS [(IN|FROM) database_name] [[LIKE] 'identifier_with_wildcards'];
+ * }}}
+ */
+case class ShowViewsCommand(
+databaseName: Option[String],
+tableIdentifierPattern: Option[String]) extends RunnableCommand {
+
+ // The result of SHOW VIEWS has three basic columns: namespace, viewName and
isTemporary.
+ override val output: Seq[Attribute] = Seq(
+AttributeReference("namespace", StringType, nullable = false)(),
+AttributeReference("viewName", StringType, nullable = false)(),
+AttributeReference("isTemporary", BooleanType, nullable = false)())
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+val catalog = sparkSession.sessionState.catalog
+val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+
+// Show the information of views.
+val views = tableIdentifierPattern.map(catalog.listViews(db, _))
+ .getOrElse(catalog.listViews(db, "*"))
+views.map { tableIdent =>
+ val namespace = tableIdent.database.getOrElse("")
Review comment:
Oh..I just force pushed from another machine ... Sorry. will updated it back.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate
SparkQA commented on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate URL: https://github.com/apache/spark/pull/28089#issuecomment-608215187 **[Test build #120741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120741/testReport)** for PR 28089 at commit [`3d12aa6`](https://github.com/apache/spark/commit/3d12aa62008e2e7eff3efab035c3c04f2f458b75). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate
SparkQA removed a comment on issue #28089: [SPARK-30921][PySpark] Predicates on python udf should not be pushdown through Aggregate URL: https://github.com/apache/spark/pull/28089#issuecomment-608149244 **[Test build #120741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120741/testReport)** for PR 28089 at commit [`3d12aa6`](https://github.com/apache/spark/commit/3d12aa62008e2e7eff3efab035c3c04f2f458b75). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins removed a comment on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-608214887 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25448/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27897: [SPARK-31113][SQL] Add SHOW VIEWS command
cloud-fan commented on a change in pull request #27897: [SPARK-31113][SQL] Add
SHOW VIEWS command
URL: https://github.com/apache/spark/pull/27897#discussion_r402722761
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -295,6 +295,41 @@ case class AlterViewAsCommand(
}
}
+/**
+ * A command for users to get views in the given database.
+ * If a databaseName is not given, the current database will be used.
+ * The syntax of using this command in SQL is:
+ * {{{
+ * SHOW VIEWS [(IN|FROM) database_name] [[LIKE] 'identifier_with_wildcards'];
+ * }}}
+ */
+case class ShowViewsCommand(
+databaseName: Option[String],
+tableIdentifierPattern: Option[String]) extends RunnableCommand {
+
+ // The result of SHOW VIEWS has three basic columns: namespace, viewName and
isTemporary.
+ override val output: Seq[Attribute] = Seq(
+AttributeReference("namespace", StringType, nullable = false)(),
+AttributeReference("viewName", StringType, nullable = false)(),
+AttributeReference("isTemporary", BooleanType, nullable = false)())
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+val catalog = sparkSession.sessionState.catalog
+val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+
+// Show the information of views.
+val views = tableIdentifierPattern.map(catalog.listViews(db, _))
+ .getOrElse(catalog.listViews(db, "*"))
+views.map { tableIdent =>
+ val namespace = tableIdent.database.getOrElse("")
Review comment:
Hmm, is this really updated?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata
AmplabJenkins commented on issue #28102: [SPARK-31327][SQL] Write Spark version into Avro file metadata URL: https://github.com/apache/spark/pull/28102#issuecomment-608214882 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
