[GitHub] [spark] SparkQA commented on issue #27434: [SPARK-30704][INFRA] Use jekyll-redirect-from 0.15.0 instead of the latest
SparkQA commented on issue #27434: [SPARK-30704][INFRA] Use jekyll-redirect-from 0.15.0 instead of the latest URL: https://github.com/apache/spark/pull/27434#issuecomment-581108860 **[Test build #117733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117733/testReport)** for PR 27434 at commit [`a06d553`](https://github.com/apache/spark/commit/a06d553ed5ba9f9e4f8bc2318874f4ce965305b2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
AmplabJenkins commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions URL: https://github.com/apache/spark/pull/27433#issuecomment-581108783 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
AmplabJenkins commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions URL: https://github.com/apache/spark/pull/27433#issuecomment-581108784 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117731/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
SparkQA commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions URL: https://github.com/apache/spark/pull/27433#issuecomment-581108776 **[Test build #117731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117731/testReport)** for PR 27433 at commit [`909bfa8`](https://github.com/apache/spark/commit/909bfa860891cc48663347f81c03607bcdf99640). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27434: [SPARK-30704][INFRA] Use jekyll-redirect-from 0.15.0 instead of the latest
AmplabJenkins commented on issue #27434: [SPARK-30704][INFRA] Use jekyll-redirect-from 0.15.0 instead of the latest URL: https://github.com/apache/spark/pull/27434#issuecomment-581108473 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22494/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27434: [SPARK-30704][INFRA] Use jekyll-redirect-from 0.15.0 instead of the latest
AmplabJenkins commented on issue #27434: [SPARK-30704][INFRA] Use jekyll-redirect-from 0.15.0 instead of the latest URL: https://github.com/apache/spark/pull/27434#issuecomment-581108472 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #27434: [SPARK-30704][INFRA] Use jekyll-redirect-from 0.15.0 instead of the latest
dongjoon-hyun opened a new pull request #27434: [SPARK-30704][INFRA] Use jekyll-redirect-from 0.15.0 instead of the latest URL: https://github.com/apache/spark/pull/27434 …atest ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-581108007 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22493/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins removed a comment on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-581108005 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-581108005 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
AmplabJenkins commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-581108007 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22493/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions.
SparkQA commented on issue #27278: [SPARK-30569][SQL][PYSPARK][SPARKR] Add percentile_approx DSL functions. URL: https://github.com/apache/spark/pull/27278#issuecomment-581107945 **[Test build #117732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117732/testReport)** for PR 27278 at commit [`83f8536`](https://github.com/apache/spark/commit/83f8536c5c205ec892ebeec72d326aabe315e0e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
AmplabJenkins removed a comment on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions URL: https://github.com/apache/spark/pull/27433#issuecomment-581106650 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution
SparkQA removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581094306 **[Test build #117725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117725/testReport)** for PR 27431 at commit [`51496bc`](https://github.com/apache/spark/commit/51496bc86579d57bfa5b2296180496924a34548f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution
AmplabJenkins removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581106706 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution
AmplabJenkins removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581106709 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117725/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT
AmplabJenkins removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT URL: https://github.com/apache/spark/pull/27428#issuecomment-581105354 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117724/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT
SparkQA removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT URL: https://github.com/apache/spark/pull/27428#issuecomment-581091759 **[Test build #117724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117724/testReport)** for PR 27428 at commit [`5c38bbe`](https://github.com/apache/spark/commit/5c38bbee893fb684980c2b10ba0442311b5cedae). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
AmplabJenkins removed a comment on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions URL: https://github.com/apache/spark/pull/27433#issuecomment-581106651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22492/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT
AmplabJenkins removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT URL: https://github.com/apache/spark/pull/27428#issuecomment-581105352 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27431: [MINOR][SQL] Improve readability for window execution
AmplabJenkins commented on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581106706 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27431: [MINOR][SQL] Improve readability for window execution
AmplabJenkins commented on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581106709 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117725/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
AmplabJenkins commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions URL: https://github.com/apache/spark/pull/27433#issuecomment-581106650 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
AmplabJenkins commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions URL: https://github.com/apache/spark/pull/27433#issuecomment-581106651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22492/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27431: [MINOR][SQL] Improve readability for window execution
SparkQA commented on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581106591 **[Test build #117725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117725/testReport)** for PR 27431 at commit [`51496bc`](https://github.com/apache/spark/commit/51496bc86579d57bfa5b2296180496924a34548f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
SparkQA commented on issue #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions URL: https://github.com/apache/spark/pull/27433#issuecomment-581106568 **[Test build #117731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117731/testReport)** for PR 27433 at commit [`909bfa8`](https://github.com/apache/spark/commit/909bfa860891cc48663347f81c03607bcdf99640). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on a change in pull request #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf
zsxwing commented on a change in pull request #27425:
[SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to
StaticSQLConf
URL: https://github.com/apache/spark/pull/27425#discussion_r373824802
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##
@@ -176,4 +176,23 @@ object StaticSQLConf {
.internal()
.booleanConf
.createWithDefault(true)
+
+ val STREAMING_UI_ENABLED =
+buildStaticConf("spark.sql.streaming.ui.enabled")
+ .doc("Whether to run the structured streaming UI for the Spark
application.")
Review comment:
nit: `Whether to run the Structured Streaming Web UI for the Spark
application when the Spark Web UI is enabled.`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on a change in pull request #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf
zsxwing commented on a change in pull request #27425:
[SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to
StaticSQLConf
URL: https://github.com/apache/spark/pull/27425#discussion_r373824863
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##
@@ -176,4 +176,23 @@ object StaticSQLConf {
.internal()
.booleanConf
.createWithDefault(true)
+
+ val STREAMING_UI_ENABLED =
+buildStaticConf("spark.sql.streaming.ui.enabled")
+ .doc("Whether to run the structured streaming UI for the Spark
application.")
+ .booleanConf
+ .createWithDefault(true)
+
+ val STREAMING_UI_RETAINED_PROGRESS_UPDATES =
+buildStaticConf("spark.sql.streaming.ui.retainedProgressUpdates")
+ .doc("The number of progress updates to retain for a streaming query for
structured " +
Review comment:
nit: `structured streaming ui` -> `Structured Streaming UI`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on a change in pull request #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf
zsxwing commented on a change in pull request #27425:
[SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to
StaticSQLConf
URL: https://github.com/apache/spark/pull/27425#discussion_r373824921
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
##
@@ -146,12 +146,14 @@ private[sql] class SharedState(
*/
lazy val streamingQueryStatusListener: Option[StreamingQueryStatusListener]
= {
val sqlConf = SQLConf.get
Review comment:
nit: we can remove this and use `conf` in this class directly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on a change in pull request #27425: [SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to StaticSQLConf
zsxwing commented on a change in pull request #27425:
[SPARK-29543][SS][FOLLOWUP] Move `spark.sql.streaming.ui.*` configs to
StaticSQLConf
URL: https://github.com/apache/spark/pull/27425#discussion_r373824898
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
##
@@ -176,4 +176,23 @@ object StaticSQLConf {
.internal()
.booleanConf
.createWithDefault(true)
+
+ val STREAMING_UI_ENABLED =
+buildStaticConf("spark.sql.streaming.ui.enabled")
+ .doc("Whether to run the structured streaming UI for the Spark
application.")
+ .booleanConf
+ .createWithDefault(true)
+
+ val STREAMING_UI_RETAINED_PROGRESS_UPDATES =
+buildStaticConf("spark.sql.streaming.ui.retainedProgressUpdates")
+ .doc("The number of progress updates to retain for a streaming query for
structured " +
+"streaming ui.")
+ .intConf
+ .createWithDefault(100)
+
+ val STREAMING_UI_RETAINED_QUERIES =
+buildStaticConf("spark.sql.streaming.ui.retainedQueries")
+ .doc("The number of inactive queries to retain for structured streaming
ui.")
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zero323 opened a new pull request #27433: [SPARK-30682][SPARKR][SQL] Add SparkR interface for higher order functions
zero323 opened a new pull request #27433: [SPARK-30682][SPARKR][SQL] Add SparkR
interface for higher order functions
URL: https://github.com/apache/spark/pull/27433
### What changes were proposed in this pull request?
This PR add R API for invoking following higher functions:
- `transform` -> `array_transform` (to avoid conflict with
`base::transform`).
- `exists` -> `array_exists` (to avoid conflict with `base::exists`).
- `forall` -> `array_forall` (no conflicts, renamed for consistency)
- `filter` -> `array_filter` (to avoid conflict with `stats::filter`).
- `aggregate` -> `array_aggregate` (to avoid conflict with
`stats::transform`).
- `zip_with` -> `arrays_zip_with` (no conflicts, renamed for consistency)
- `transform_keys`
- `transform_values`
- `map_filter`
- `map_zip_with`
Overall implementation follows the same pattern as proposed for PySpark
(#27406) and reuses object supporting Scala implementation (SPARK-27297).
### Why are the changes needed?
Currently higher order functions are available only using SQL and Scala API
and can use only SQL expressions:
```r
select(df, expr("transform(xs, x -> x + 1)")
```
This is error-prone, and hard to do right, when complex logic is used
(`when` / `otherwise`, complex objects).
If this PR is accepted, above function could simply rewritten as:
```r
select(df, transform("xs", function(x) x + 1))
```
### Does this PR introduce any user-facing change?
No (but new user-facing functions are added).
### How was this patch tested?
Added new unit tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT
AmplabJenkins commented on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT URL: https://github.com/apache/spark/pull/27428#issuecomment-581105352 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT
AmplabJenkins commented on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT URL: https://github.com/apache/spark/pull/27428#issuecomment-581105354 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117724/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT
SparkQA commented on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT URL: https://github.com/apache/spark/pull/27428#issuecomment-581105270 **[Test build #117724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117724/testReport)** for PR 27428 at commit [`5c38bbe`](https://github.com/apache/spark/commit/5c38bbee893fb684980c2b10ba0442311b5cedae). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104201 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117727/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104198 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104115 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117728/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104198 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104201 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117727/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
SparkQA removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097469 **[Test build #117727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117727/testReport)** for PR 27207 at commit [`2924c5b`](https://github.com/apache/spark/commit/2924c5b908d3e05295fd0c6101f746dd5dfa2e9d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
SparkQA commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104167 **[Test build #117727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117727/testReport)** for PR 27207 at commit [`2924c5b`](https://github.com/apache/spark/commit/2924c5b908d3e05295fd0c6101f746dd5dfa2e9d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104113 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
SparkQA removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097846 **[Test build #117728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117728/testReport)** for PR 27207 at commit [`e0ac12e`](https://github.com/apache/spark/commit/e0ac12e77bffe0f6f1c3592b9bcda0c7e0dbf046). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104113 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104115 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117728/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
SparkQA commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581104073 **[Test build #117728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117728/testReport)** for PR 27207 at commit [`e0ac12e`](https://github.com/apache/spark/commit/e0ac12e77bffe0f6f1c3592b9bcda0c7e0dbf046). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581103835 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117729/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
SparkQA removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581100363 **[Test build #117729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117729/testReport)** for PR 27432 at commit [`865bcc6`](https://github.com/apache/spark/commit/865bcc622cfb3934cb32f918c683ee062950f420). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581103833 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581103833 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581103835 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117729/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
SparkQA commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581103810 **[Test build #117729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117729/testReport)** for PR 27432 at commit [`865bcc6`](https://github.com/apache/spark/commit/865bcc622cfb3934cb32f918c683ee062950f420). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class EscapeRegexExpression(str: Expression, pattern: Expression, escape: Expression)` * `case class Like(str: Expression, pattern: Expression, escape: Expression)` * `case class SimilarTo(str: Expression, pattern: Expression, escape: Expression)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott edited a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott edited a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581102206 @tgravescs I updated with the new config to switch to legacy behavior as well as some doc/variable renaming. I also updated the description of this PR. After reading more through `SchedulerBackend` impls: 1. decided to assume resource offers are full resource by default, to match previous behavior for any `SchedulerBackend`s not touched in this PR. 2. I only found `MesosFineGrainedSchedulerBackend` which doesn't use `CoarseGrainedSchedulerBackend` and decided not to touch it since it is deprecated and from the design looks like "all free resources" can't be tracked. Test updates are still needed, but wanted to get your feedback first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott edited a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott edited a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581102206 @tgravescs I updated with the new config to switch to legacy behavior as well as some doc/variable renaming. After reading more through `SchedulerBackend` impls: 1. decided to assume resource offers are full resource by default, to match previous behavior for any `SchedulerBackend`s not touched in this PR. 2. I only found `MesosFineGrainedSchedulerBackend` which doesn't use `CoarseGrainedSchedulerBackend` and decided not to touch it since it is deprecated and from the design looks like "all free resources" can't be tracked. Test updates are still needed, but wanted to get your feedback first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581102206 @tgravescs I updated with the new config to switch to legacy behavior as well as some doc/variable renaming. After reading more through `SchedulerBackend` impls: 1. decided to assume resource offers are full resource by default, to match previous behavior for any `SchedulerBackend`s not touched in this PR. 2. I only found `MesosFineGrainedSchedulerBackend` which doesn't use `CoarseGrainedSchedulerBackend` and decided not to touch it since it is deprecated and from the design looks like "all free resources" can't be tracked. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott commented on a change in pull request #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on a change in pull request #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#discussion_r373821397 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ## @@ -100,6 +101,11 @@ private[spark] class TaskSchedulerImpl( // on this class. Protected by `this` private val taskSetsByStageIdAndAttempt = new HashMap[Int, HashMap[Int, TaskSetManager]] + // keyed by task set stage id + // value is true if there have been no resources rejected due to delay scheduling + // since the last "full" resource offer + private val noDelayScheduleRejects = new mutable.HashMap[Int, Boolean]() Review comment: what do you see as pros/cons of keeping this map vs putting in TSM? The `TaskSchedulerImpl` is the only one setting/updating the variable based on knowledge only it has (no schedule delay reject). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
SparkQA commented on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27429#issuecomment-581101682 **[Test build #117730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117730/testReport)** for PR 27429 at commit [`4f22a7a`](https://github.com/apache/spark/commit/4f22a7a18192e084267243bf6854434292c06c1c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins commented on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27429#issuecomment-581101353 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22491/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins removed a comment on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27429#issuecomment-581101350 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins removed a comment on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27429#issuecomment-581101353 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22491/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression
AmplabJenkins commented on issue #27429: [SPARK-28330][SQL] Support ANSI SQL: result offset clause in query expression URL: https://github.com/apache/spark/pull/27429#issuecomment-581101350 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581100442 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581100442 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581100444 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22490/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
AmplabJenkins removed a comment on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581100444 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22490/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
SparkQA commented on issue #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432#issuecomment-581100363 **[Test build #117729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117729/testReport)** for PR 27432 at commit [`865bcc6`](https://github.com/apache/spark/commit/865bcc622cfb3934cb32f918c683ee062950f420). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer closed pull request #27007: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
beliefer closed pull request #27007: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27007 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer opened a new pull request #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax
beliefer opened a new pull request #27432: [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax URL: https://github.com/apache/spark/pull/27432 ### What changes were proposed in this pull request? `[ NOT ] SIMILAR TO [ ESCAPE ]` This is a ANSI SQL. For example: ``` SELECT 'abc' SIMILAR TO 'abc' AS `true`; SELECT 'abc' SIMILAR TO 'a' AS `false`; SELECT 'abc' SIMILAR TO '%(b|d)%' AS `true`; SELECT 'abc' SIMILAR TO '(b|c)%' AS `false`; ``` There are some mainstream database support the syntax. **PostgreSQL:** https://www.postgresql.org/docs/current/functions-matching.html#FUNCTIONS-SIMILARTO-REGEXP **Redshift:** https://docs.aws.amazon.com/redshift/latest/dg/pattern-matching-conditions-similar-to.html `SIMILAR TO` is similar to `RLIKE`, but with the following differences: 1. The `SIMILAR TO` operator returns true only if its pattern matches the entire string, unlike `RLIKE` behavior, where the pattern can match any portion of the string. 2. The regex string allow uses _ and % as wildcard characters denoting any single character and any string, respectively (these are comparable to . and .* in POSIX regular expressions). 3. The regex string allow uses escape character like `LIKE` behavior. 4. '.', '^' and '$' is not a meta character for `SIMILAR TO`. ### Why are the changes needed? Support ANSI SQL: `SIMILAR TO ... ESCAPE` syntax `SIMILAR TO ... ESCAPE ` has a lot of convenience compared to `RLIKE`. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Exists and new UT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
dongjoon-hyun closed pull request #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097962 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22489/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097958 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097962 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22489/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097958 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
SparkQA commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097846 **[Test build #117728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117728/testReport)** for PR 27207 at commit [`e0ac12e`](https://github.com/apache/spark/commit/e0ac12e77bffe0f6f1c3592b9bcda0c7e0dbf046). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097527 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097528 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22488/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097527 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
AmplabJenkins removed a comment on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097528 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22488/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
SparkQA commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-581097469 **[Test build #117727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117727/testReport)** for PR 27207 at commit [`2924c5b`](https://github.com/apache/spark/commit/2924c5b908d3e05295fd0c6101f746dd5dfa2e9d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
AmplabJenkins removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581097215 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
AmplabJenkins commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581097217 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117726/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
SparkQA removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581096736 **[Test build #117726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117726/testReport)** for PR 27161 at commit [`99dfd0d`](https://github.com/apache/spark/commit/99dfd0da0068518903d6ece007ef52ea8468955a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
AmplabJenkins removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581097217 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117726/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
AmplabJenkins commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581097215 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
SparkQA commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581097202 **[Test build #117726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117726/testReport)** for PR 27161 at commit [`99dfd0d`](https://github.com/apache/spark/commit/99dfd0da0068518903d6ece007ef52ea8468955a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
AmplabJenkins removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581096785 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22487/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
AmplabJenkins removed a comment on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581096783 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
AmplabJenkins commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581096785 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22487/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
AmplabJenkins commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581096783 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
SparkQA commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581096736 **[Test build #117726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117726/testReport)** for PR 27161 at commit [`99dfd0d`](https://github.com/apache/spark/commit/99dfd0da0068518903d6ece007ef52ea8468955a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
gatorsmile commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#discussion_r373817637 ## File path: sql/core/src/test/resources/sql-tests/results/interval.sql.out ## @@ -1101,3 +1102,45 @@ select interval '1 ' day struct -- !query 113 output 1 days + + +-- !query 114 +select -(a) from values (interval '-2147483648 months', interval '2147483647 months') t(a, b) +-- !query 114 schema +struct<(- a):interval> +-- !query 114 output +-178956970 years -8 months + + +-- !query 115 +select a - b from values (interval '-2147483648 months', interval '2147483647 months') t(a, b) +-- !query 115 schema +struct<(a - b):interval> +-- !query 115 output +1 months + + +-- !query 116 +select b + interval '1 month' from values (interval '-2147483648 months', interval '2147483647 months') t(a, b) +-- !query 116 schema +struct<(b + INTERVAL '1 months'):interval> +-- !query 116 output +-178956970 years -8 months + + +-- !query 117 +select a * 1.1 from values (interval '-2147483648 months', interval '2147483647 months') t(a, b) +-- !query 117 schema +struct<> +-- !query 117 output +java.lang.ArithmeticException +integer overflow Review comment: It sounds like the option 2 looks good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default
dongjoon-hyun commented on issue #27161: [SPARK-27686][DOC][SQL] Update migration guide for make Hive 2.3 dependency by default URL: https://github.com/apache/spark/pull/27161#issuecomment-581094762 Could you resolve the conflicts, @wangyum ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution
AmplabJenkins removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581094354 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT
AmplabJenkins removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT URL: https://github.com/apache/spark/pull/27428#issuecomment-581091879 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution
AmplabJenkins removed a comment on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581094356 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22486/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT
AmplabJenkins removed a comment on issue #27428: [SPARK-30276][SQL] Support Filter expression allows simultaneous use of DISTINCT URL: https://github.com/apache/spark/pull/27428#issuecomment-581091881 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22485/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27431: [MINOR][SQL] Improve readability for window execution
AmplabJenkins commented on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581094356 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22486/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27431: [MINOR][SQL] Improve readability for window execution
AmplabJenkins commented on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581094354 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27431: [MINOR][SQL] Improve readability for window execution
SparkQA commented on issue #27431: [MINOR][SQL] Improve readability for window execution URL: https://github.com/apache/spark/pull/27431#issuecomment-581094306 **[Test build #117725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117725/testReport)** for PR 27431 at commit [`51496bc`](https://github.com/apache/spark/commit/51496bc86579d57bfa5b2296180496924a34548f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #27381: [MINOR][SQL] Improve readability for some code comments
beliefer commented on issue #27381: [MINOR][SQL] Improve readability for some code comments URL: https://github.com/apache/spark/pull/27381#issuecomment-581094111 This PR migrate to https://github.com/apache/spark/pull/27431 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
