[GitHub] [spark] SparkQA commented on pull request #31190: [SPARK-34118][CORE][SQL][3.0] Replaces filter and check for emptiness with exists or forall
SparkQA commented on pull request #31190: URL: https://github.com/apache/spark/pull/31190#issuecomment-760720166 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38672/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 closed pull request #31053: [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions
Ngone51 closed pull request #31053: URL: https://github.com/apache/spark/pull/31053 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31107: [SPARK-34052][SQL] A cached view should become invalid after the source table is dropped
sunchao commented on a change in pull request #31107: URL: https://github.com/apache/spark/pull/31107#discussion_r557982546 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -833,6 +833,7 @@ case class CacheTable( case class CacheTableAsSelect( tempViewName: String, plan: LogicalPlan, +originalText: Option[String], Review comment: I only see `AstBuilder` as the only source for constructing a `CacheTableAsSelect` so yes we can probably remove the `Option`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31107: [SPARK-34052][SQL] A cached view should become invalid after the source table is dropped
sunchao commented on a change in pull request #31107:
URL: https://github.com/apache/spark/pull/31107#discussion_r557980220
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala
##
@@ -399,7 +399,7 @@ class CatalogImpl(sparkSession: SparkSession) extends
Catalog {
try {
val plan = sparkSession.sessionState.executePlan(viewDef)
sparkSession.sharedState.cacheManager.uncacheQuery(
- sparkSession, plan.analyzed, cascade = false)
+ sparkSession, plan.analyzed, cascade = true)
Review comment:
Oops forgot to update here. Will update and add test cases.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31107: [SPARK-34052][SQL] A cached view should become invalid after the source table is dropped
sunchao commented on a change in pull request #31107:
URL: https://github.com/apache/spark/pull/31107#discussion_r557979466
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##
@@ -238,7 +238,7 @@ case class DropTableCommand(
if (isTempView || catalog.tableExists(tableName)) {
try {
sparkSession.sharedState.cacheManager.uncacheQuery(
- sparkSession.table(tableName), cascade = !isTempView)
+ sparkSession.table(tableName), cascade = !(isTempView &&
conf.storeAnalyzedPlanForView))
Review comment:
Good point. Will update.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30974: [SPARK-33939][SQL] Make Column.named UnresolvedExtractValue use UnresolvedAlias to assign name
SparkQA commented on pull request #30974: URL: https://github.com/apache/spark/pull/30974#issuecomment-760717657 **[Test build #134093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134093/testReport)** for PR 30974 at commit [`65fe331`](https://github.com/apache/spark/commit/65fe331d0ec609232a2b5a92622753531181ac66). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31186: [SPARK-33790][CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
SparkQA commented on pull request #31186: URL: https://github.com/apache/spark/pull/31186#issuecomment-760717442 **[Test build #134091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134091/testReport)** for PR 31186 at commit [`41c2277`](https://github.com/apache/spark/commit/41c22773f7b4ffe7805bd443f35940013b976146). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31160: [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector
SparkQA commented on pull request #31160: URL: https://github.com/apache/spark/pull/31160#issuecomment-760717455 **[Test build #134092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134092/testReport)** for PR 31160 at commit [`1a9d26b`](https://github.com/apache/spark/commit/1a9d26bd99437c54ce6fd229ed141c897972a82e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
SparkQA commented on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760713933 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38673/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31160: [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector
AmplabJenkins removed a comment on pull request #31160: URL: https://github.com/apache/spark/pull/31160#issuecomment-760713222 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38668/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31186: [SPARK-33790][CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
AmplabJenkins removed a comment on pull request #31186: URL: https://github.com/apache/spark/pull/31186#issuecomment-760713219 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134075/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
AmplabJenkins removed a comment on pull request #31187: URL: https://github.com/apache/spark/pull/31187#issuecomment-760713231 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38667/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31142: [SPARK-33346][CORE][SQL][MLLIB][DSTREAM][K8S] Change the never changed 'var' to 'val'
AmplabJenkins removed a comment on pull request #31142: URL: https://github.com/apache/spark/pull/31142#issuecomment-760713232 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134077/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29737: [SPARK-32864][SQL] Support ORC forced positional evolution
AmplabJenkins removed a comment on pull request #29737: URL: https://github.com/apache/spark/pull/29737#issuecomment-760713228 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134070/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31192: [SPARK-34118][CORE][SQL][2.4] Replaces filter and check for emptiness with exists or forall
AmplabJenkins removed a comment on pull request #31192: URL: https://github.com/apache/spark/pull/31192#issuecomment-760713223 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38671/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31142: [SPARK-33346][CORE][SQL][MLLIB][DSTREAM][K8S] Change the never changed 'var' to 'val'
AmplabJenkins commented on pull request #31142: URL: https://github.com/apache/spark/pull/31142#issuecomment-760713232 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134077/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31189: [SPARK-34122][SQL] Remove duplicated branches in CaseWhen
AmplabJenkins removed a comment on pull request #31189: URL: https://github.com/apache/spark/pull/31189#issuecomment-760713218 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38665/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31160: [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector
AmplabJenkins commented on pull request #31160: URL: https://github.com/apache/spark/pull/31160#issuecomment-760713222 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38668/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31192: [SPARK-34118][CORE][SQL][2.4] Replaces filter and check for emptiness with exists or forall
AmplabJenkins commented on pull request #31192: URL: https://github.com/apache/spark/pull/31192#issuecomment-760713223 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38671/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
AmplabJenkins commented on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760713224 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134076/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31188: [SPARK-34118][CORE][SQL][3.1] Replaces filter and check for emptiness with exists or forall
AmplabJenkins commented on pull request #31188: URL: https://github.com/apache/spark/pull/31188#issuecomment-760713220 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38666/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
AmplabJenkins commented on pull request #31187: URL: https://github.com/apache/spark/pull/31187#issuecomment-760713231 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38667/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31188: [SPARK-34118][CORE][SQL][3.1] Replaces filter and check for emptiness with exists or forall
AmplabJenkins removed a comment on pull request #31188: URL: https://github.com/apache/spark/pull/31188#issuecomment-760713220 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38666/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
AmplabJenkins removed a comment on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760713224 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134076/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31193: [SPARK-34117][SQL] Disable LeftSemi/LeftAnti push down over Aggregate
AmplabJenkins removed a comment on pull request #31193: URL: https://github.com/apache/spark/pull/31193#issuecomment-760690501 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31189: [SPARK-34122][SQL] Remove duplicated branches in CaseWhen
AmplabJenkins commented on pull request #31189: URL: https://github.com/apache/spark/pull/31189#issuecomment-760713218 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38665/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31193: [SPARK-34117][SQL] Disable LeftSemi/LeftAnti push down over Aggregate
AmplabJenkins commented on pull request #31193: URL: https://github.com/apache/spark/pull/31193#issuecomment-760713230 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38670/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29737: [SPARK-32864][SQL] Support ORC forced positional evolution
AmplabJenkins commented on pull request #29737: URL: https://github.com/apache/spark/pull/29737#issuecomment-760713228 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134070/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31186: [SPARK-33790][CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
AmplabJenkins commented on pull request #31186: URL: https://github.com/apache/spark/pull/31186#issuecomment-760713219 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134075/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
SparkQA commented on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760713017 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38676/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31193: [SPARK-34117][SQL] Disable LeftSemi/LeftAnti push down over Aggregate
SparkQA commented on pull request #31193: URL: https://github.com/apache/spark/pull/31193#issuecomment-760712164 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38670/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31079: [SPARK-34037][SQL] Remove unnecessary upcasting for Avg & Sum which handle by themself internally
yaooqinn commented on pull request #31079: URL: https://github.com/apache/spark/pull/31079#issuecomment-760711076 updated thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31079: [SPARK-34037][SQL] Remove unnecessary upcasting for Avg & Sum which handle by themself internally
SparkQA commented on pull request #31079: URL: https://github.com/apache/spark/pull/31079#issuecomment-760710848 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38675/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31193: [SPARK-34117][SQL] Disable LeftSemi/LeftAnti push down over Aggregate
SparkQA commented on pull request #31193: URL: https://github.com/apache/spark/pull/31193#issuecomment-760710298 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38670/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31142: [SPARK-33346][CORE][SQL][MLLIB][DSTREAM][K8S] Change the never changed 'var' to 'val'
SparkQA removed a comment on pull request #31142: URL: https://github.com/apache/spark/pull/31142#issuecomment-760643095 **[Test build #134077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134077/testReport)** for PR 31142 at commit [`c599a05`](https://github.com/apache/spark/commit/c599a054b0ae30ce75a359331ebc9bde366ceac2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31190: [SPARK-34118][CORE][SQL][3.0] Replaces filter and check for emptiness with exists or forall
SparkQA commented on pull request #31190: URL: https://github.com/apache/spark/pull/31190#issuecomment-760709735 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38672/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31142: [SPARK-33346][CORE][SQL][MLLIB][DSTREAM][K8S] Change the never changed 'var' to 'val'
SparkQA commented on pull request #31142: URL: https://github.com/apache/spark/pull/31142#issuecomment-760709225 **[Test build #134077 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134077/testReport)** for PR 31142 at commit [`c599a05`](https://github.com/apache/spark/commit/c599a054b0ae30ce75a359331ebc9bde366ceac2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31079: [SPARK-34037][SQL] ResolveAggregateFunctions pushes duplicated sort order into aggregate because of unnecessary casting
cloud-fan commented on pull request #31079: URL: https://github.com/apache/spark/pull/31079#issuecomment-760705381 @yaooqinn can we update the PR title and description? Now it's a simple improvement to avoid unnecessary casts for sum/avg. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31186: [SPARK-33790][CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
HeartSaVioR commented on pull request #31186: URL: https://github.com/apache/spark/pull/31186#issuecomment-760704537 retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31119: [SPARK-34064][SQL] Cancel the running broadcast sub-jobs when SQL statement is cancelled
cloud-fan commented on a change in pull request #31119:
URL: https://github.com/apache/spark/pull/31119#discussion_r557936826
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
##
@@ -74,7 +74,10 @@ case class BroadcastExchangeExec(
child: SparkPlan) extends BroadcastExchangeLike {
import BroadcastExchangeExec._
- override val runId: UUID = UUID.randomUUID
+ // Cancelling a SQL statement from Spark ThriftServer needs to cancel
+ // its related broadcast sub-jobs. So set the run id to job group id if
exists.
+ override val runId: UUID =
Option(sparkContext.getLocalProperty(SparkContext.SPARK_JOB_GROUP_ID))
Review comment:
@LantaoJin what do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31119: [SPARK-34064][SQL] Cancel the running broadcast sub-jobs when SQL statement is cancelled
cloud-fan commented on a change in pull request #31119:
URL: https://github.com/apache/spark/pull/31119#discussion_r557936604
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
##
@@ -74,7 +74,10 @@ case class BroadcastExchangeExec(
child: SparkPlan) extends BroadcastExchangeLike {
import BroadcastExchangeExec._
- override val runId: UUID = UUID.randomUUID
+ // Cancelling a SQL statement from Spark ThriftServer needs to cancel
+ // its related broadcast sub-jobs. So set the run id to job group id if
exists.
+ override val runId: UUID =
Option(sparkContext.getLocalProperty(SparkContext.SPARK_JOB_GROUP_ID))
Review comment:
After a second thought, I think this is risky. It's possible that in a
non-STS environment, users set job group id manually, and run some long-running
jobs. If we capture the job group id here in broadcast exchange, when the
broadcast timeout, it will cancel the whole job group which may kill the user's
other long-running jobs unexpectedly.
I think we need to revisit the STS's SQL statement canceling feature. We
should use SQL execution ID to find out all the jobs of a SQL query, and assign
a unique job group id to them.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31192: [SPARK-34118][CORE][SQL][2.4] Replaces filter and check for emptiness with exists or forall
SparkQA commented on pull request #31192: URL: https://github.com/apache/spark/pull/31192#issuecomment-760703741 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38671/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30746: [SPARK-33766][BUILD] Upgrade Jackson to 2.11.4
LuciferYang commented on pull request #30746: URL: https://github.com/apache/spark/pull/30746#issuecomment-760703656 Got it, I'll file a new JIRA and try to fix it first :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
SparkQA removed a comment on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760641904 **[Test build #134076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134076/testReport)** for PR 31185 at commit [`f8f12a0`](https://github.com/apache/spark/commit/f8f12a0734ac2708d41ea68a8721b85541c6d4ca). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
SparkQA commented on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760702444 **[Test build #134076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134076/testReport)** for PR 31185 at commit [`f8f12a0`](https://github.com/apache/spark/commit/f8f12a0734ac2708d41ea68a8721b85541c6d4ca). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31107: [SPARK-34052][SQL] A cached view should become invalid after the source table is dropped
viirya commented on a change in pull request #31107: URL: https://github.com/apache/spark/pull/31107#discussion_r557932490 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -833,6 +833,7 @@ case class CacheTable( case class CacheTableAsSelect( tempViewName: String, plan: LogicalPlan, +originalText: Option[String], Review comment: Is there any other source of `CacheTableAsSelect` other than `AstBuilder`? Don't we always have this `originalText`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #31107: [SPARK-34052][SQL] A cached view should become invalid after the source table is dropped
viirya commented on a change in pull request #31107: URL: https://github.com/apache/spark/pull/31107#discussion_r557932490 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -833,6 +833,7 @@ case class CacheTable( case class CacheTableAsSelect( tempViewName: String, plan: LogicalPlan, +originalText: Option[String], Review comment: Is any other source of `CacheTableAsSelect` other than `AstBuilder`? Don't we always have this `originalText`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31186: [SPARK-33790][CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
SparkQA removed a comment on pull request #31186: URL: https://github.com/apache/spark/pull/31186#issuecomment-760641870 **[Test build #134075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134075/testReport)** for PR 31186 at commit [`41c2277`](https://github.com/apache/spark/commit/41c22773f7b4ffe7805bd443f35940013b976146). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31186: [SPARK-33790][CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
SparkQA commented on pull request #31186: URL: https://github.com/apache/spark/pull/31186#issuecomment-760701574 **[Test build #134075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134075/testReport)** for PR 31186 at commit [`41c2277`](https://github.com/apache/spark/commit/41c22773f7b4ffe7805bd443f35940013b976146). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29737: [SPARK-32864][SQL] Support ORC forced positional evolution
SparkQA removed a comment on pull request #29737: URL: https://github.com/apache/spark/pull/29737#issuecomment-760607924 **[Test build #134070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134070/testReport)** for PR 29737 at commit [`51f503c`](https://github.com/apache/spark/commit/51f503c5738a714d6ea77467ac8f7dfba231d989). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29737: [SPARK-32864][SQL] Support ORC forced positional evolution
SparkQA commented on pull request #29737: URL: https://github.com/apache/spark/pull/29737#issuecomment-760700894 **[Test build #134070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134070/testReport)** for PR 29737 at commit [`51f503c`](https://github.com/apache/spark/commit/51f503c5738a714d6ea77467ac8f7dfba231d989). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31160: [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector
SparkQA commented on pull request #31160: URL: https://github.com/apache/spark/pull/31160#issuecomment-760699987 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38668/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31193: [SPARK-34117][SQL] Disable LeftSemi/LeftAnti push down over Aggregate
cloud-fan commented on pull request #31193: URL: https://github.com/apache/spark/pull/31193#issuecomment-760699379 why do we need to disable it completely? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31192: [SPARK-34118][CORE][SQL][2.4] Replaces filter and check for emptiness with exists or forall
SparkQA commented on pull request #31192: URL: https://github.com/apache/spark/pull/31192#issuecomment-760698666 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38671/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
SparkQA commented on pull request #31187: URL: https://github.com/apache/spark/pull/31187#issuecomment-760698590 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38667/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31188: [SPARK-34118][CORE][SQL][3.1] Replaces filter and check for emptiness with exists or forall
SparkQA commented on pull request #31188: URL: https://github.com/apache/spark/pull/31188#issuecomment-760697233 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38666/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
SparkQA commented on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760695785 **[Test build #134090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134090/testReport)** for PR 31185 at commit [`f8f12a0`](https://github.com/apache/spark/commit/f8f12a0734ac2708d41ea68a8721b85541c6d4ca). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #30508: [SPARK-33618][CORE] Use hadoop-client instead of hadoop-client-api to make hadoop-aws work
yaooqinn commented on a change in pull request #30508: URL: https://github.com/apache/spark/pull/30508#discussion_r557919211 ## File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3 ## @@ -116,14 +142,30 @@ jline/2.14.6//jline-2.14.6.jar joda-time/2.10.5//joda-time-2.10.5.jar jodd-core/3.5.2//jodd-core-3.5.2.jar jpam/1.1//jpam-1.1.jar +json-smart/2.3//json-smart-2.3.jar json/1.8//json-1.8.jar json4s-ast_2.12/3.7.0-M5//json4s-ast_2.12-3.7.0-M5.jar json4s-core_2.12/3.7.0-M5//json4s-core_2.12-3.7.0-M5.jar json4s-jackson_2.12/3.7.0-M5//json4s-jackson_2.12-3.7.0-M5.jar json4s-scalap_2.12/3.7.0-M5//json4s-scalap_2.12-3.7.0-M5.jar +jsp-api/2.1//jsp-api-2.1.jar jsr305/3.0.0//jsr305-3.0.0.jar jta/1.1//jta-1.1.jar jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar +kerb-admin/1.0.1//kerb-admin-1.0.1.jar +kerb-client/1.0.1//kerb-client-1.0.1.jar +kerb-common/1.0.1//kerb-common-1.0.1.jar +kerb-core/1.0.1//kerb-core-1.0.1.jar +kerb-crypto/1.0.1//kerb-crypto-1.0.1.jar +kerb-identity/1.0.1//kerb-identity-1.0.1.jar +kerb-server/1.0.1//kerb-server-1.0.1.jar +kerb-simplekdc/1.0.1//kerb-simplekdc-1.0.1.jar Review comment: looks like the original PR does not handle any transitive artifact exclusion at all 😸 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31189: [SPARK-34122][SQL] Remove duplicated branches in CaseWhen
SparkQA commented on pull request #31189: URL: https://github.com/apache/spark/pull/31189#issuecomment-760694192 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38665/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #30974: [SPARK-33939][SQL] Make Column.named UnresolvedExtractValue use UnresolvedAlias to assign name
ulysses-you commented on pull request #30974: URL: https://github.com/apache/spark/pull/30974#issuecomment-760693554 Seems there is another issue, `Literal(null, NullType)` will be convert to other data type during analysis. Then `Null` will be `CAST(Null AS xxx)`. https://github.com/apache/spark/blob/00d43b1f829fb5f79f0355afbbacc804162648e5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala#L390-L392 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
HyukjinKwon commented on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760692189 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30746: [SPARK-33766][BUILD] Upgrade Jackson to 2.11.4
HyukjinKwon commented on pull request #30746: URL: https://github.com/apache/spark/pull/30746#issuecomment-760691842 Yes, we should. Not sure if we'll have more 2.4.x releases though: https://spark.apache.org/versioning-policy.html > The last minor release within a major a release will typically be maintained for longer as an “LTS” release. For example, 2.4.0 was released in November 2018, but will likely see releases for more than 18 months, beyond May 2020. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
AmplabJenkins removed a comment on pull request #31187: URL: https://github.com/apache/spark/pull/31187#issuecomment-760690502 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134081/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31190: [SPARK-34118][CORE][SQL][3.0] Replaces filter and check for emptiness with exists or forall
AmplabJenkins removed a comment on pull request #31190: URL: https://github.com/apache/spark/pull/31190#issuecomment-760690500 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
AmplabJenkins commented on pull request #31187: URL: https://github.com/apache/spark/pull/31187#issuecomment-760690502 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134081/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31193: [SPARK-34117][SQL] Disable LeftSemi/LeftAnti push down over Aggregate
AmplabJenkins commented on pull request #31193: URL: https://github.com/apache/spark/pull/31193#issuecomment-760690501 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134085/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31190: [SPARK-34118][CORE][SQL][3.0] Replaces filter and check for emptiness with exists or forall
AmplabJenkins commented on pull request #31190: URL: https://github.com/apache/spark/pull/31190#issuecomment-760690500 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30508: [SPARK-33618][CORE] Use hadoop-client instead of hadoop-client-api to make hadoop-aws work
HyukjinKwon commented on a change in pull request #30508: URL: https://github.com/apache/spark/pull/30508#discussion_r557909634 ## File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3 ## @@ -116,14 +142,30 @@ jline/2.14.6//jline-2.14.6.jar joda-time/2.10.5//joda-time-2.10.5.jar jodd-core/3.5.2//jodd-core-3.5.2.jar jpam/1.1//jpam-1.1.jar +json-smart/2.3//json-smart-2.3.jar json/1.8//json-1.8.jar json4s-ast_2.12/3.7.0-M5//json4s-ast_2.12-3.7.0-M5.jar json4s-core_2.12/3.7.0-M5//json4s-core_2.12-3.7.0-M5.jar json4s-jackson_2.12/3.7.0-M5//json4s-jackson_2.12-3.7.0-M5.jar json4s-scalap_2.12/3.7.0-M5//json4s-scalap_2.12-3.7.0-M5.jar +jsp-api/2.1//jsp-api-2.1.jar jsr305/3.0.0//jsr305-3.0.0.jar jta/1.1//jta-1.1.jar jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar +kerb-admin/1.0.1//kerb-admin-1.0.1.jar +kerb-client/1.0.1//kerb-client-1.0.1.jar +kerb-common/1.0.1//kerb-common-1.0.1.jar +kerb-core/1.0.1//kerb-core-1.0.1.jar +kerb-crypto/1.0.1//kerb-crypto-1.0.1.jar +kerb-identity/1.0.1//kerb-identity-1.0.1.jar +kerb-server/1.0.1//kerb-server-1.0.1.jar +kerb-simplekdc/1.0.1//kerb-simplekdc-1.0.1.jar Review comment: This is actually a revert. It was added in https://github.com/apache/spark/commit/ce7ba2e98e0a3b038e881c271b5905058c43155b#diff-e45e1eee8dcfd7eaf8a013cec02b67806da3edeabe0f195ac6b4402f67d4b6dcR146 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31079: [SPARK-34037][SQL] ResolveAggregateFunctions pushes duplicated sort order into aggregate because of unnecessary casting
SparkQA commented on pull request #31079: URL: https://github.com/apache/spark/pull/31079#issuecomment-760690112 **[Test build #134089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134089/testReport)** for PR 31079 at commit [`e891f5e`](https://github.com/apache/spark/commit/e891f5e0640ae78491d76826b43ae6b2edd20e64). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31190: [SPARK-34118][CORE][SQL][3.0] Replaces filter and check for emptiness with exists or forall
SparkQA commented on pull request #31190: URL: https://github.com/apache/spark/pull/31190#issuecomment-760690020 **[Test build #134087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134087/testReport)** for PR 31190 at commit [`4ac192b`](https://github.com/apache/spark/commit/4ac192bd30d9bc0d54330789d9b3b0694203b8df). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31192: [SPARK-34118][CORE][SQL][2.4] Replaces filter and check for emptiness with exists or forall
SparkQA commented on pull request #31192: URL: https://github.com/apache/spark/pull/31192#issuecomment-760690017 **[Test build #134086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134086/testReport)** for PR 31192 at commit [`e309061`](https://github.com/apache/spark/commit/e309061820be1746a82914b3b45dc579b969e7b8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31142: [SPARK-33346][CORE][SQL][MLLIB][DSTREAM][K8S] Change the never changed 'var' to 'val'
SparkQA commented on pull request #31142: URL: https://github.com/apache/spark/pull/31142#issuecomment-760690046 **[Test build #134088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134088/testReport)** for PR 31142 at commit [`c599a05`](https://github.com/apache/spark/commit/c599a054b0ae30ce75a359331ebc9bde366ceac2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31187: [SPARK-33790][CORE][3.0] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
SparkQA commented on pull request #31187: URL: https://github.com/apache/spark/pull/31187#issuecomment-760689350 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38667/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
yaooqinn commented on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760689228 ``` - decommission workers ensure that shuffle output is regenerated even with shuffle service *** FAILED *** (20 seconds, 783 milliseconds) [info] 5 did not equal 4 Expected 4 tasks but got List(0:0:0:0-SUCCESS, 0:0:1:0-FAILED, 0:0:1:1-SUCCESS, 0:1:0:0-SUCCESS, 1:0:0:0-SUCCESS) (DecommissionWorkerSuite.scala:190) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) [info] at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231) [info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295) [info] at org.apache.spark.deploy.DecommissionWorkerSuite.$anonfun$new$6(DecommissionWorkerSuite.scala:190) ``` seems just a flaky test itself and not related to this change This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31053: [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions
AmplabJenkins removed a comment on pull request #31053: URL: https://github.com/apache/spark/pull/31053#issuecomment-760688678 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38669/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31191: [SPARK-34123][Web UI] optimize spark history summary page loading
AmplabJenkins commented on pull request #31191: URL: https://github.com/apache/spark/pull/31191#issuecomment-760688769 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31053: [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions
AmplabJenkins commented on pull request #31053: URL: https://github.com/apache/spark/pull/31053#issuecomment-760688678 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38669/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31053: [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions
SparkQA commented on pull request #31053: URL: https://github.com/apache/spark/pull/31053#issuecomment-760688662 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38669/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
AmplabJenkins removed a comment on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760688348 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38662/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31160: [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector
AmplabJenkins removed a comment on pull request #31160: URL: https://github.com/apache/spark/pull/31160#issuecomment-760688346 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134082/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31186: [SPARK-33790][CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
AmplabJenkins removed a comment on pull request #31186: URL: https://github.com/apache/spark/pull/31186#issuecomment-760688349 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38661/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30974: [SPARK-33939][SQL] Make Column.named UnresolvedExtractValue use UnresolvedAlias to assign name
AmplabJenkins removed a comment on pull request #30974: URL: https://github.com/apache/spark/pull/30974#issuecomment-760688345 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134072/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31142: [SPARK-33346][CORE][SQL][MLLIB][DSTREAM][K8S] Change the never changed 'var' to 'val'
AmplabJenkins removed a comment on pull request #31142: URL: https://github.com/apache/spark/pull/31142#issuecomment-760688347 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38663/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31160: [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector
AmplabJenkins commented on pull request #31160: URL: https://github.com/apache/spark/pull/31160#issuecomment-760688346 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134082/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31142: [SPARK-33346][CORE][SQL][MLLIB][DSTREAM][K8S] Change the never changed 'var' to 'val'
AmplabJenkins commented on pull request #31142: URL: https://github.com/apache/spark/pull/31142#issuecomment-760688347 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38663/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31186: [SPARK-33790][CORE][3.1] Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
AmplabJenkins commented on pull request #31186: URL: https://github.com/apache/spark/pull/31186#issuecomment-760688349 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38661/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31185: [SPARK-34111][BUILD][test-maven] Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
AmplabJenkins commented on pull request #31185: URL: https://github.com/apache/spark/pull/31185#issuecomment-760688348 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38662/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30974: [SPARK-33939][SQL] Make Column.named UnresolvedExtractValue use UnresolvedAlias to assign name
AmplabJenkins commented on pull request #30974: URL: https://github.com/apache/spark/pull/30974#issuecomment-760688345 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134072/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31053: [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions
SparkQA commented on pull request #31053: URL: https://github.com/apache/spark/pull/31053#issuecomment-760686545 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38669/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30746: [SPARK-33766][BUILD] Upgrade Jackson to 2.11.4
LuciferYang commented on pull request #30746: URL: https://github.com/apache/spark/pull/30746#issuecomment-760686077 ``` FasterXML jackson-databind 2.x before 2.9.10.8 mishandles the interaction between serialization gadgets and typing, related to oadd.org.apache.commons.dbcp.cpdsadapter.DriverAdapterCPDS. ``` [CVE-2020-36179](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-36179) I found Spark 2.4 still using Jackson 2.6.7 https://github.com/apache/spark/blob/63e93a5c38a83669ccc58a5b45d5cff0b296fcc9/pom.xml#L161-L163 Should we upgrade it and which version should we use in Spark 2.4. cc @wangyum @dongjoon-hyun @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum opened a new pull request #31193: [SPARK-34117][SQL] Disable LeftSemi/LeftAnti push down over Aggregate
wangyum opened a new pull request #31193: URL: https://github.com/apache/spark/pull/31193 ### What changes were proposed in this pull request? This pr disable LeftSemi/LeftAnti push down over Aggregate. ### Why are the changes needed? LeftSemi/LeftAnti push down over Aggregate may affect performance. for example: ```sql SELECT i_item_sk ss_item_sk FROM item, (SELECT distinct iss.i_brand_id brand_id, iss.i_class_id class_id, iss.i_category_id category_id FROM store_sales, item iss, date_dim d1 WHERE ss_item_sk = iss.i_item_sk AND ss_sold_date_sk = d1.d_date_sk AND d1.d_year BETWEEN 1999 AND 1999 + 2 INTERSECT SELECT distinct ics.i_brand_id, ics.i_class_id, ics.i_category_id FROM catalog_sales, item ics, date_dim d2 WHERE cs_item_sk = ics.i_item_sk AND cs_sold_date_sk = d2.d_date_sk AND d2.d_year BETWEEN 1999 AND 1999 + 2 INTERSECT SELECT distinct iws.i_brand_id, iws.i_class_id, iws.i_category_id FROM web_sales, item iws, date_dim d3 WHERE ws_item_sk = iws.i_item_sk AND ws_sold_date_sk = d3.d_date_sk AND d3.d_year BETWEEN 1999 AND 1999 + 2) x WHERE i_brand_id = brand_id AND i_class_id = class_id AND i_category_id = category_id; ``` This query is rewritten from [q14b](https://github.com/apache/spark/blob/a78d6ce376edf2a8836e01f47b9dff5371058d4c/sql/core/src/test/resources/tpcds/q14b.sql#L2-L32). CBO enabled | CBO disabled -- | -- The statistics are more accurate, so it will be converted to broadcast join, and then push down aggregate. | It will not push down aggregate.  |  ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31066: [SPARK-34027][SQL] Refresh cache in `ALTER TABLE .. RECOVER PARTITIONS`
MaxGekk commented on pull request #31066: URL: https://github.com/apache/spark/pull/31066#issuecomment-760685706 @HyukjinKwon @cloud-fan Please, take a look at this PR one more time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31160: [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector
SparkQA removed a comment on pull request #31160: URL: https://github.com/apache/spark/pull/31160#issuecomment-760661019 **[Test build #134082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134082/testReport)** for PR 31160 at commit [`d8601e7`](https://github.com/apache/spark/commit/d8601e7cdf06904e7d5450e87292da8c5cb37036). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31160: [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector
SparkQA commented on pull request #31160: URL: https://github.com/apache/spark/pull/31160#issuecomment-760685496 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38668/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31160: [SPARK-34080][ML][PYTHON] Add UnivariateFeatureSelector
SparkQA commented on pull request #31160: URL: https://github.com/apache/spark/pull/31160#issuecomment-760685337 **[Test build #134082 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134082/testReport)** for PR 31160 at commit [`d8601e7`](https://github.com/apache/spark/commit/d8601e7cdf06904e7d5450e87292da8c5cb37036). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class _UnivariateFeatureSelectorParams(HasFeaturesCol, HasOutputCol, HasLabelCol):` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31188: [SPARK-34118][CORE][SQL][3.1] Replaces filter and check for emptiness with exists or forall
SparkQA commented on pull request #31188: URL: https://github.com/apache/spark/pull/31188#issuecomment-760684233 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38666/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31190: [SPARK-34118][CORE][SQL][3.0] Replaces filter and check for emptiness with exists or forall
LuciferYang commented on a change in pull request #31190:
URL: https://github.com/apache/spark/pull/31190#discussion_r557896600
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryListenerSuite.scala
##
@@ -48,8 +48,7 @@ class StreamingQueryListenerSuite extends StreamTest with
BeforeAndAfter {
spark.streams.active.foreach(_.stop())
assert(spark.streams.active.isEmpty)
// Skip check default `StreamingQueryStatusListener` which is for
streaming UI.
-assert(spark.streams.listListeners()
- .filterNot(_.isInstanceOf[StreamingQueryStatusListener]).isEmpty)
+
assert(spark.streams.listListeners().forall(_.isInstanceOf[StreamingQueryStatusListener]))
Review comment:
Address 4ac192b add a newline
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31189: [SPARK-34122][SQL] Remove duplicated branches in CaseWhen
SparkQA commented on pull request #31189: URL: https://github.com/apache/spark/pull/31189#issuecomment-760682298 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38665/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #30508: [SPARK-33618][CORE] Use hadoop-client instead of hadoop-client-api to make hadoop-aws work
yaooqinn commented on a change in pull request #30508: URL: https://github.com/apache/spark/pull/30508#discussion_r557892419 ## File path: dev/deps/spark-deps-hadoop-3.2-hive-2.3 ## @@ -116,14 +142,30 @@ jline/2.14.6//jline-2.14.6.jar joda-time/2.10.5//joda-time-2.10.5.jar jodd-core/3.5.2//jodd-core-3.5.2.jar jpam/1.1//jpam-1.1.jar +json-smart/2.3//json-smart-2.3.jar json/1.8//json-1.8.jar json4s-ast_2.12/3.7.0-M5//json4s-ast_2.12-3.7.0-M5.jar json4s-core_2.12/3.7.0-M5//json4s-core_2.12-3.7.0-M5.jar json4s-jackson_2.12/3.7.0-M5//json4s-jackson_2.12-3.7.0-M5.jar json4s-scalap_2.12/3.7.0-M5//json4s-scalap_2.12-3.7.0-M5.jar +jsp-api/2.1//jsp-api-2.1.jar jsr305/3.0.0//jsr305-3.0.0.jar jta/1.1//jta-1.1.jar jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar +kerb-admin/1.0.1//kerb-admin-1.0.1.jar +kerb-client/1.0.1//kerb-client-1.0.1.jar +kerb-common/1.0.1//kerb-common-1.0.1.jar +kerb-core/1.0.1//kerb-core-1.0.1.jar +kerb-crypto/1.0.1//kerb-crypto-1.0.1.jar +kerb-identity/1.0.1//kerb-identity-1.0.1.jar +kerb-server/1.0.1//kerb-server-1.0.1.jar +kerb-simplekdc/1.0.1//kerb-simplekdc-1.0.1.jar Review comment: just for curiosity, does spark has a chance to play the role of KDC at runtime? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31190: [SPARK-34118][CORE][SQL][3.0] Replaces filter and check for emptiness with exists or forall
LuciferYang commented on a change in pull request #31190:
URL: https://github.com/apache/spark/pull/31190#discussion_r557892183
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryListenerSuite.scala
##
@@ -48,8 +48,7 @@ class StreamingQueryListenerSuite extends StreamTest with
BeforeAndAfter {
spark.streams.active.foreach(_.stop())
assert(spark.streams.active.isEmpty)
// Skip check default `StreamingQueryStatusListener` which is for
streaming UI.
-assert(spark.streams.listListeners()
- .filterNot(_.isInstanceOf[StreamingQueryStatusListener]).isEmpty)
+
assert(spark.streams.listListeners().forall(_.isInstanceOf[StreamingQueryStatusListener]))
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
