[GitHub] [spark] AmplabJenkins commented on pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

AmplabJenkins commented on pull request #31638: URL: https://github.com/apache/spark/pull/31638#issuecomment-802038630 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40794/ -- T

[GitHub] [spark] SparkQA commented on pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

SparkQA commented on pull request #31638: URL: https://github.com/apache/spark/pull/31638#issuecomment-802036977 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40794/ -- This is an automated message from the A

[GitHub] [spark] HyukjinKwon commented on pull request #31851: [SPARK-34760][EXAMPLES] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

HyukjinKwon commented on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-802033111 👏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] SparkQA commented on pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

SparkQA commented on pull request #31638: URL: https://github.com/apache/spark/pull/31638#issuecomment-802030050 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40794/ -- This is an automated message from the Apache

[GitHub] [spark] yaooqinn commented on pull request #31851: [SPARK-34760][EXAMPLES] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

yaooqinn commented on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-802011948 @zengruios Thanks for your first contribution to Apache Spark. I have added you as a contributor at the JIRA side, and SPARK-34760 has been assigned to you. Thanks

[GitHub] [spark] AmplabJenkins commented on pull request #31883: [MINOR][DOCS][ML] Doc 'mode' as a supported Imputer strategy in Pyspark

AmplabJenkins commented on pull request #31883: URL: https://github.com/apache/spark/pull/31883#issuecomment-802005355 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40792/ -- T

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31877: [SPARK-34783][K8S] Support remote template files

dongjoon-hyun edited a comment on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-802002261 Thank you, @attilapiros . I'm still looking at the code to find some workaround. I'll comment on this PR later if I find a better solution. -- This is an aut

[GitHub] [spark] AmplabJenkins commented on pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

AmplabJenkins commented on pull request #31854: URL: https://github.com/apache/spark/pull/31854#issuecomment-802005427 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40793/ -- T

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

AmplabJenkins removed a comment on pull request #31638: URL: https://github.com/apache/spark/pull/31638#issuecomment-802004242 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136212/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

AmplabJenkins commented on pull request #31638: URL: https://github.com/apache/spark/pull/31638#issuecomment-802004242 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136212/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #31883: [MINOR][DOCS][ML] Doc 'mode' as a supported Imputer strategy in Pyspark

AmplabJenkins commented on pull request #31883: URL: https://github.com/apache/spark/pull/31883#issuecomment-802003969 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136210/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

AmplabJenkins commented on pull request #31854: URL: https://github.com/apache/spark/pull/31854#issuecomment-802004075 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136211/ -- This

[GitHub] [spark] dongjoon-hyun commented on pull request #31877: [SPARK-34783][K8S] Support remote template files

dongjoon-hyun commented on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-802002261 Thank you, @attilapiros . I'm still looking at the code to find some workaround. I'll comment on this PR later if I found a better solution. -- This is an automated

[GitHub] [spark] attilapiros commented on pull request #31877: [SPARK-34783][K8S] Support remote template files

attilapiros commented on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-801999551 > For the test case concern, I can still remove it but I prefer to have the test coverage for the new function for now. We can revise the test suite later with the `with

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31877: [SPARK-34783][K8S] Support remote template files

dongjoon-hyun edited a comment on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-801995607 @HyukjinKwon , @attilapiros , @viirya . I tried to use `DependencyUtils`, but it turns out that it fails at the execution because `Utils.doFetchFile` depends on `

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31877: [SPARK-34783][K8S] Support remote template files

dongjoon-hyun edited a comment on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-801995607 @HyukjinKwon , @attilapiros , @viirya . I tried to use `DependencyUtils`, but it turns out that it fails at the execution because `Utils.doFetchFile` depends on `

[GitHub] [spark] yaooqinn commented on pull request #31851: [SPARK-34760][EXAMPLES] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

yaooqinn commented on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-801996284 My network is not in good condition at the moment. It took years to fetch and push this PR to master :(.. Now, it's fighting for branch-3.1 -- This is an automated message f

[GitHub] [spark] dongjoon-hyun commented on pull request #31877: [SPARK-34783][K8S] Support remote template files

dongjoon-hyun commented on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-801995607 @HyukjinKwon , @attilapiros , @viirya . I tried to use `DependencyUtils`, but it turns out that it fails at the execution because `Utils.doFetchFile` depends on `SparkEn

[GitHub] [spark] yaooqinn closed pull request #31851: [SPARK-34760][EXAMPLES] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

yaooqinn closed pull request #31851: URL: https://github.com/apache/spark/pull/31851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, pleas

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

AmplabJenkins removed a comment on pull request #31880: URL: https://github.com/apache/spark/pull/31880#issuecomment-801708633 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] AmplabJenkins commented on pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

AmplabJenkins commented on pull request #31880: URL: https://github.com/apache/spark/pull/31880#issuecomment-801991520 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40791/ -- T

[GitHub] [spark] bersprockets commented on pull request #31826: [SPARK-34731][CORE] Avoid ConcurrentModificationException when redacting properties in EventLoggingListener

bersprockets commented on pull request #31826: URL: https://github.com/apache/spark/pull/31826#issuecomment-801988869 Thanks @HeartSaVioR @HyukjinKwon ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] attilapiros commented on a change in pull request #31877: [SPARK-34783][K8S] Support remote template files

attilapiros commented on a change in pull request #31877: URL: https://github.com/apache/spark/pull/31877#discussion_r596933300 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala ## @@ -79,12 +79,25 @@ private[spark

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31877: [SPARK-34783][K8S] Support remote template files

dongjoon-hyun commented on a change in pull request #31877: URL: https://github.com/apache/spark/pull/31877#discussion_r596930356 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala ## @@ -79,12 +79,25 @@ private[spa

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31877: [SPARK-34783][K8S] Support remote template files

dongjoon-hyun commented on a change in pull request #31877: URL: https://github.com/apache/spark/pull/31877#discussion_r596929133 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala ## @@ -79,12 +79,25 @@ private[spa

[GitHub] [spark] sarutak commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

sarutak commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596919236 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: The first time I op

[GitHub] [spark] sarutak commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

sarutak commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596919236 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: The first time I op

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31877: [SPARK-34783][K8S] Support remote template files

dongjoon-hyun commented on a change in pull request #31877: URL: https://github.com/apache/spark/pull/31877#discussion_r596927422 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesUtilsSuite.scala ## @@ -65,4 +65,14 @@ class K

[GitHub] [spark] xuanyuanking commented on a change in pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

xuanyuanking commented on a change in pull request #31638: URL: https://github.com/apache/spark/pull/31638#discussion_r596927316 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala ## @@ -40,17 +41,31 @@ object FileStreamSink ext

[GitHub] [spark] cloud-fan commented on a change in pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

cloud-fan commented on a change in pull request #31859: URL: https://github.com/apache/spark/pull/31859#discussion_r596926825 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercionSuite.scala ## @@ -377,10 +368,26 @@ class AnsiTypeCoe

[GitHub] [spark] SparkQA commented on pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

SparkQA commented on pull request #31880: URL: https://github.com/apache/spark/pull/31880#issuecomment-801976512 **[Test build #136209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136209/testReport)** for PR 31880 at commit [`3c66069`](https://github.com

[GitHub] [spark] gengliangwang commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

gengliangwang commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-801976239 > Is there any function using TypeCollection[Long, Double]? There are similar expressions: Divide: `TypeCollection(DoubleType, DecimalType)` IntegralDivide:

[GitHub] [spark] srowen opened a new pull request #31883: [MINOR][DOCS][ML] Doc 'mode' as a supported Imputer strategy in Pyspark

srowen opened a new pull request #31883: URL: https://github.com/apache/spark/pull/31883 ### What changes were proposed in this pull request? Document `mode` as a supported Imputer strategy in Pyspark docs. ### Why are the changes needed? Support was added in 3.1, and do

[GitHub] [spark] xuanyuanking commented on a change in pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

xuanyuanking commented on a change in pull request #31638: URL: https://github.com/apache/spark/pull/31638#discussion_r596922892 ## File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala ## @@ -575,6 +576,43 @@ abstract class FileStreamSink

[GitHub] [spark] xuanyuanking commented on a change in pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

xuanyuanking commented on a change in pull request #31638: URL: https://github.com/apache/spark/pull/31638#discussion_r596922353 ## File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala ## @@ -575,6 +576,43 @@ abstract class FileStreamSink

[GitHub] [spark] xuanyuanking commented on a change in pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

xuanyuanking commented on a change in pull request #31638: URL: https://github.com/apache/spark/pull/31638#discussion_r596922117 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala ## @@ -40,17 +41,31 @@ object FileStreamSink ext

[GitHub] [spark] xuanyuanking commented on a change in pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

xuanyuanking commented on a change in pull request #31638: URL: https://github.com/apache/spark/pull/31638#discussion_r596921444 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala ## @@ -40,17 +41,31 @@ object FileStreamSink ext

[GitHub] [spark] sarutak commented on a change in pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

sarutak commented on a change in pull request #31854: URL: https://github.com/apache/spark/pull/31854#discussion_r596920385 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala ## @@ -206,8 +206,13 @@ object UnresolvedAttribute {

[GitHub] [spark] sarutak commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

sarutak commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596919236 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: The first time I op

[GitHub] [spark] yaooqinn commented on pull request #31851: [SPARK-34760][EXAMPLES][MINOR] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

yaooqinn commented on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-801965701 Yea, LGTM~ This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [spark] cloud-fan commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

cloud-fan commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-801964936 Is there any function using `TypeCollection[Long, Double]`? This is an automated message from the Apache Git S

[GitHub] [spark] HyukjinKwon commented on pull request #31851: [SPARK-34760][EXAMPLES][MINOR] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

HyukjinKwon commented on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-801963789 Improvements are not backported in general but looks like this is a bug fix in the example (reading from JIRA) which is usually backported. The JIRA states the affected vers

[GitHub] [spark] cloud-fan commented on a change in pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

cloud-fan commented on a change in pull request #31854: URL: https://github.com/apache/spark/pull/31854#discussion_r596911804 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala ## @@ -206,8 +206,13 @@ object UnresolvedAttribute {

[GitHub] [spark] yaooqinn commented on pull request #31851: [SPARK-34760][EXAMPLES][MINOR] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

yaooqinn commented on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-801961498 thanks, @HyukjinKwon. I will merge this to master only, is it Okay? This is an automated message from the Apac

[GitHub] [spark] HyukjinKwon commented on pull request #31851: [SPARK-34760][EXAMPLES][MINOR] Replace `favorite_color` with `age` in JavaSQLDataSourceExample

HyukjinKwon commented on pull request #31851: URL: https://github.com/apache/spark/pull/31851#issuecomment-801958904 @yaooqinn maybe can you try merging this as a brand new committer :-)? This is an automated message from the

[GitHub] [spark] sarutak commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

sarutak commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596900664 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: It's to preserve on

[GitHub] [spark] srowen commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

srowen commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596904316 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: I don't see the adva

[GitHub] [spark] sarutak commented on pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

sarutak commented on pull request #31854: URL: https://github.com/apache/spark/pull/31854#issuecomment-801955964 cc: @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] gengliangwang commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

gengliangwang commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-801955243 > Shall we simply fail if there are multiple matches in the type collection? It's a bit tricky to define "closest" That sounds not reasonable. E.g., failing an input

[GitHub] [spark] sarutak commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

sarutak commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596900664 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: It's to preserve on

[GitHub] [spark] sarutak commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

sarutak commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596900664 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: It's to preserve on

[GitHub] [spark] AmplabJenkins commented on pull request #31882: [WIP][SPARK-34787] Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

AmplabJenkins commented on pull request #31882: URL: https://github.com/apache/spark/pull/31882#issuecomment-801946594 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] luhenry commented on pull request #30810: [SPARK-33882][ML] Add a vectorized BLAS implementation

luhenry commented on pull request #30810: URL: https://github.com/apache/spark/pull/30810#issuecomment-801944250 Hi, as JDK 16 has been released, I’ll follow up on this in the upcoming days to check if it’s still building and passing the test suite, and fix any possible regressions.

[GitHub] [spark] cloud-fan edited a comment on pull request #31653: [SPARK-33832][SQL] v2. move OptimzieSkewedJoin to query stage preparation

cloud-fan edited a comment on pull request #31653: URL: https://github.com/apache/spark/pull/31653#issuecomment-801934545 > and I need to know which ones were added by OptimizeSkewedJoin and which were added by something else. Sorry, I may miss something. This PR puts `OptimizeSkewed

[GitHub] [spark] cloud-fan edited a comment on pull request #31653: [SPARK-33832][SQL] v2. move OptimzieSkewedJoin to query stage preparation

cloud-fan edited a comment on pull request #31653: URL: https://github.com/apache/spark/pull/31653#issuecomment-801934545 > and I need to know which ones were added by OptimizeSkewedJoin and which were added by something else. Sorry, I may miss something. This PR puts `OptimizeSkewed

[GitHub] [spark] cloud-fan commented on pull request #31653: [SPARK-33832][SQL] v2. move OptimzieSkewedJoin to query stage preparation

cloud-fan commented on pull request #31653: URL: https://github.com/apache/spark/pull/31653#issuecomment-801934545 > and I need to know which ones were added by OptimizeSkewedJoin and which were added by something else. Sorry, I may miss something. This PR puts `OptimizeSkewedJoin` i

[GitHub] [spark] ulysses-you commented on pull request #31879: [SPARK-34766][SQL][3.1] Do not capture maven config for views

ulysses-you commented on pull request #31879: URL: https://github.com/apache/spark/pull/31879#issuecomment-801934177 thanks for merging! This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [spark] cloud-fan commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

cloud-fan commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-801928537 Shall we simply fail if there are multiple matches in the type collection? It's a bit tricky to define "closest" -

[GitHub] [spark] kyoty opened a new pull request #31882: [WIP][SPARK-34787] Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

kyoty opened a new pull request #31882: URL: https://github.com/apache/spark/pull/31882 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was t

[GitHub] [spark] Ngone51 commented on a change in pull request #31869: [SPARK-34777][UI] StagePage input size records not show when records greater than zero

Ngone51 commented on a change in pull request #31869: URL: https://github.com/apache/spark/pull/31869#discussion_r596858929 ## File path: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ## @@ -786,9 +786,13 @@ private[spark] object ApiHelper { stageData.accumu

[GitHub] [spark] cloud-fan commented on a change in pull request #31756: [SPARK-34637] [SQL] [WIP] Support DPP when the broadcast exchange can be reused

cloud-fan commented on a change in pull request #31756: URL: https://github.com/apache/spark/pull/31756#discussion_r596855011 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala ## @@ -41,15 +42,26 @@ case class

[GitHub] [spark] AmplabJenkins commented on pull request #31470: [SPARK-34354][SQL] Fix failure when apply CostBasedJoinReorder on self-join

AmplabJenkins commented on pull request #31470: URL: https://github.com/apache/spark/pull/31470#issuecomment-801913222 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136208/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31878: [SPARK-34784][BUILD] Upgrade Jackson to 2.12.2

AmplabJenkins removed a comment on pull request #31878: URL: https://github.com/apache/spark/pull/31878#issuecomment-801912907 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136207/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31878: [SPARK-34784][BUILD] Upgrade Jackson to 2.12.2

AmplabJenkins commented on pull request #31878: URL: https://github.com/apache/spark/pull/31878#issuecomment-801912907 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136207/ -

[GitHub] [spark] cloud-fan commented on a change in pull request #31756: [SPARK-34637] [SQL] [WIP] Support DPP when the broadcast exchange can be reused

cloud-fan commented on a change in pull request #31756: URL: https://github.com/apache/spark/pull/31756#discussion_r596850988 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala ## @@ -41,15 +42,26 @@ case class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31470: [SPARK-34354][SQL] Fix failure when apply CostBasedJoinReorder on self-join

AmplabJenkins removed a comment on pull request #31470: URL: https://github.com/apache/spark/pull/31470#issuecomment-801911973 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40790/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31878: [SPARK-34784][BUILD] Upgrade Jackson to 2.12.2

AmplabJenkins removed a comment on pull request #31878: URL: https://github.com/apache/spark/pull/31878#issuecomment-801745499 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #31470: [SPARK-34354][SQL] Fix failure when apply CostBasedJoinReorder on self-join

AmplabJenkins commented on pull request #31470: URL: https://github.com/apache/spark/pull/31470#issuecomment-801911973 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40790/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31878: [SPARK-34784][BUILD] Upgrade Jackson to 2.12.2

AmplabJenkins commented on pull request #31878: URL: https://github.com/apache/spark/pull/31878#issuecomment-801911659 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40789/ -

[GitHub] [spark] Ngone51 closed pull request #31881: [SPARK-34087][3.1][SQL] Fix memory leak of ExecutionListenerBus

Ngone51 closed pull request #31881: URL: https://github.com/apache/spark/pull/31881 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] Ngone51 commented on a change in pull request #31842: [SPARK-34748][SS] Create a rule of the analysis logic for streaming write

Ngone51 commented on a change in pull request #31842: URL: https://github.com/apache/spark/pull/31842#discussion_r596834842 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala ## @@ -0,0 +1,99 @@ +/* + * Licensed to the Apa

[GitHub] [spark] srowen commented on a change in pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible

srowen commented on a change in pull request #31588: URL: https://github.com/apache/spark/pull/31588#discussion_r596846214 ## File path: mllib/src/main/scala/org/apache/spark/ml/feature/VectorSlicer.scala ## @@ -110,22 +110,21 @@ final class VectorSlicer @Since("1.5.0") (@Since

[GitHub] [spark] srowen commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

srowen commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596845664 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: I don't quite see wh

[GitHub] [spark] sarutak commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

sarutak commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596843488 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: I think it's better

[GitHub] [spark] wangyum commented on a change in pull request #31857: [SPARK-28220][SQL] Push down the foldable predicate to both sides of Join

wangyum commented on a change in pull request #31857: URL: https://github.com/apache/spark/pull/31857#discussion_r596841781 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala ## @@ -1384,4 +1384,14 @@ class FilterPushdow

[GitHub] [spark] cloud-fan commented on a change in pull request #31857: [SPARK-28220][SQL] Push down the foldable predicate to both sides of Join

cloud-fan commented on a change in pull request #31857: URL: https://github.com/apache/spark/pull/31857#discussion_r596838355 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala ## @@ -1384,4 +1384,14 @@ class FilterPushd

[GitHub] [spark] cloud-fan commented on a change in pull request #31848: [SPARK-34756][SQL] Fix FileScan equality check

cloud-fan commented on a change in pull request #31848: URL: https://github.com/apache/spark/pull/31848#discussion_r596837272 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala ## @@ -86,7 +86,7 @@ trait FileScan extends Scan

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31842: [SPARK-34748][SS] Create a rule of the analysis logic for streaming write

AmplabJenkins removed a comment on pull request #31842: URL: https://github.com/apache/spark/pull/31842#issuecomment-801898823 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #31842: [SPARK-34748][SS] Create a rule of the analysis logic for streaming write

AmplabJenkins commented on pull request #31842: URL: https://github.com/apache/spark/pull/31842#issuecomment-801898825 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] cloud-fan closed pull request #31879: [SPARK-34766][SQL][3.1] Do not capture maven config for views

cloud-fan closed pull request #31879: URL: https://github.com/apache/spark/pull/31879 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] cloud-fan commented on pull request #31879: [SPARK-34766][SQL][3.1] Do not capture maven config for views

cloud-fan commented on pull request #31879: URL: https://github.com/apache/spark/pull/31879#issuecomment-801897535 thanks, merging to 3.1! This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [spark] LuciferYang commented on pull request #31865: [SPARK-34774][BUILD] Ensure change-scala-version.sh update scala.version in parent POM correctly

LuciferYang commented on pull request #31865: URL: https://github.com/apache/spark/pull/31865#issuecomment-801896899 > @LuciferYang do you want to make a similar change for 2.x? I can do it too. @srowen OK, I will do this later ~ -

[GitHub] [spark] attilapiros commented on a change in pull request #31877: [SPARK-34783][K8S] Support remote template files

attilapiros commented on a change in pull request #31877: URL: https://github.com/apache/spark/pull/31877#discussion_r596831803 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesUtilsSuite.scala ## @@ -65,4 +65,14 @@ class Kub

[GitHub] [spark] srowen commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

srowen commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r596828376 ## File path: pom.xml ## @@ -162,7 +162,9 @@ 3.4.1 3.2.2 -2.12.10 +2.12.10 +2.13.5 Review comment: Hm, why do we need b

[GitHub] [spark] srowen commented on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

srowen commented on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-801891092 I don't really know enough to evaluate this. It looks reasonable and tests pass. Are there any compatibility concerns, or possible changes to behavior you can think of? ---

[GitHub] [spark] srowen commented on pull request #31878: [SPARK-34784][BUILD] Upgrade Jackson to 2.12.2

srowen commented on pull request #31878: URL: https://github.com/apache/spark/pull/31878#issuecomment-801890213 Jenkins test this please This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [spark] srowen commented on pull request #31865: [SPARK-34774][BUILD] Ensure change-scala-version.sh update scala.version in parent POM correctly

srowen commented on pull request #31865: URL: https://github.com/apache/spark/pull/31865#issuecomment-801889689 (Jenkins won't test this anyway, note) Merged to master / 3.1 / 3.0. @LuciferYang do you want to make a similar change for 2.x? I can do it too. ---

[GitHub] [spark] srowen closed pull request #31865: [SPARK-34774][BUILD] Ensure change-scala-version.sh update scala.version in parent POM correctly

srowen closed pull request #31865: URL: https://github.com/apache/spark/pull/31865 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31842: [SPARK-34748][SS] Create a rule of the analysis logic for streaming write

AmplabJenkins removed a comment on pull request #31842: URL: https://github.com/apache/spark/pull/31842#issuecomment-799624243 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] xuanyuanking commented on pull request #31842: [SPARK-34748][SS] Create a rule of the analysis logic for streaming write

xuanyuanking commented on pull request #31842: URL: https://github.com/apache/spark/pull/31842#issuecomment-801863216 retest this please This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [spark] sarutak commented on pull request #31841: [SPARK-34747][SQL][DOCS] Add virtual operators to the built-in function document

sarutak commented on pull request #31841: URL: https://github.com/apache/spark/pull/31841#issuecomment-801827763 Build failure on GA is not related to this change. If there is no more comment or objection, I'll merge this tomorrow. ---

[GitHub] [spark] attilapiros commented on pull request #31876: [WIP][SPARK-XXXX][API][CORE] Abstract Location in MapStatus to enable support for custom storage

attilapiros commented on pull request #31876: URL: https://github.com/apache/spark/pull/31876#issuecomment-801809190 @Ngone51 thanks for doing this. I am currently busy with some other stuffs but next week I will do the review. -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

AmplabJenkins removed a comment on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-801793829 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136205/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

AmplabJenkins commented on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-801793829 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136205/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

AmplabJenkins removed a comment on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-801792506 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40787/

[GitHub] [spark] AmplabJenkins commented on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

AmplabJenkins commented on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-801792506 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40787/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

AmplabJenkins commented on pull request #31830: URL: https://github.com/apache/spark/pull/31830#issuecomment-801756193 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40786/ -

[GitHub] [spark] sadhen commented on a change in pull request #31735: [SPARK-34600][PYTHON][SQL] Return User-defined types from Pandas UDF

sadhen commented on a change in pull request #31735: URL: https://github.com/apache/spark/pull/31735#discussion_r596669193 ## File path: python/pyspark/sql/pandas/types.py ## @@ -74,6 +74,8 @@ def to_arrow_type(dt): arrow_type = pa.struct(fields) elif type(dt) ==

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31822: [SPARK-34728][SQL] Remove all SQLConf.get if extends from SQLConfHelper

AmplabJenkins removed a comment on pull request #31822: URL: https://github.com/apache/spark/pull/31822#issuecomment-801744634 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136193/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

AmplabJenkins commented on pull request #31830: URL: https://github.com/apache/spark/pull/31830#issuecomment-801755108 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136204/ -

<    1   2   3   4   5   >