[GitHub] [spark] AmplabJenkins commented on pull request #37591: [SPARK-40158][SQL] Remove useless configuration & extract common code for parquet read

AmplabJenkins commented on PR #37591: URL: https://github.com/apache/spark/pull/37591#issuecomment-1221462899 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #37592: [SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 2, 32 functions)

AmplabJenkins commented on PR #37592: URL: https://github.com/apache/spark/pull/37592#issuecomment-1221462892 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #37594: [SPARK-40159][SQL] Aggregate should be group only after collapse project to aggregate

AmplabJenkins commented on PR #37594: URL: https://github.com/apache/spark/pull/37594#issuecomment-1221462886 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] wankunde opened a new pull request, #37594: [SPARK-40159][SQL] Aggregate should be group only after collapse project to aggregate

wankunde opened a new pull request, #37594: URL: https://github.com/apache/spark/pull/37594 ### What changes were proposed in this pull request? Aggregate should be group only if there not exists AggregateFunction in aggregate.groupingExpressions ### Why are the

[GitHub] [spark] panbingkun commented on a diff in pull request #37591: [SPARK-40158][SQL] Remove useless configuration & extract common code for parquet read

panbingkun commented on code in PR #37591: URL: https://github.com/apache/spark/pull/37591#discussion_r950777846 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala: ## @@ -199,33 +199,7 @@ class ParquetFileFormat

[GitHub] [spark] panbingkun commented on a diff in pull request #37591: [SPARK-40158][SQL] Remove useless configuration & extract common code for parquet read

panbingkun commented on code in PR #37591: URL: https://github.com/apache/spark/pull/37591#discussion_r950777842 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala: ## @@ -199,33 +199,7 @@ class ParquetFileFormat

[GitHub] [spark] srowen commented on pull request #37478: [SPARK][SQL] feat: SparkSession.confing(Map)

srowen commented on PR #37478: URL: https://github.com/apache/spark/pull/37478#issuecomment-1221452645 You can file a JIRA, please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #37474: [SPARK-40039][SS] Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

HeartSaVioR commented on code in PR #37474: URL: https://github.com/apache/spark/pull/37474#discussion_r950767972 ## sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/CheckpointFileManagerSuite.scala: ## @@ -58,50 +77,40 @@ abstract class

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37592: [SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 2, 32 functions)

HyukjinKwon commented on code in PR #37592: URL: https://github.com/apache/spark/pull/37592#discussion_r950766376 ## python/pyspark/sql/functions.py: ## @@ -6158,12 +6764,14 @@ def _test() -> None: import doctest from pyspark.sql import Row, SparkSession import

[GitHub] [spark] HyukjinKwon commented on pull request #37592: [SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 2, 32 functions)

HyukjinKwon commented on PR #37592: URL: https://github.com/apache/spark/pull/37592#issuecomment-1221438918 Thanks for doing this, @khalidmammadov -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37591: [MINOR] Remove useless configuration & extract common code

HyukjinKwon commented on code in PR #37591: URL: https://github.com/apache/spark/pull/37591#discussion_r950766153 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala: ## @@ -199,33 +199,7 @@ class ParquetFileFormat

[GitHub] [spark] HyukjinKwon closed pull request #37582: [SPARK-40147][PYTHON][SQL] Make pyspark.sql.session examples self-contained

HyukjinKwon closed pull request #37582: [SPARK-40147][PYTHON][SQL] Make pyspark.sql.session examples self-contained URL: https://github.com/apache/spark/pull/37582 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] HyukjinKwon commented on pull request #37582: [SPARK-40147][PYTHON][SQL] Make pyspark.sql.session examples self-contained

HyukjinKwon commented on PR #37582: URL: https://github.com/apache/spark/pull/37582#issuecomment-1221438513 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon closed pull request #37593: [SPARK-40111][PYTHON][DOCS] Make pyspark.rdd examples self-contained

HyukjinKwon closed pull request #37593: [SPARK-40111][PYTHON][DOCS] Make pyspark.rdd examples self-contained URL: https://github.com/apache/spark/pull/37593 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] HyukjinKwon commented on pull request #37593: [SPARK-40111][PYTHON][DOCS] Make pyspark.rdd examples self-contained

HyukjinKwon commented on PR #37593: URL: https://github.com/apache/spark/pull/37593#issuecomment-1221438483 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37593: [SPARK-40111][PYTHON][DOCS] Make pyspark.rdd examples self-contained

HyukjinKwon commented on code in PR #37593: URL: https://github.com/apache/spark/pull/37593#discussion_r950765838 ## python/pyspark/rdd.py: ## @@ -400,9 +501,41 @@ def unpersist(self: "RDD[T]", blocking: bool = False) -> "RDD[T]": Mark the RDD as non-persistent, and

[GitHub] [spark] HyukjinKwon commented on pull request #37587: [SPARK-40081][PYTHON][DOCS] Add document parameters for pyspark.sql.streaming.query

HyukjinKwon commented on PR #37587: URL: https://github.com/apache/spark/pull/37587#issuecomment-1221436109 cc @HeartSaVioR if you find some time to take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] HyukjinKwon commented on pull request #37593: [SPARK-40111][PYTHON][DOCS] Make pyspark.rdd examples self-contained

HyukjinKwon commented on PR #37593: URL: https://github.com/apache/spark/pull/37593#issuecomment-1221436039 Wow, thank you @zhengruifeng for doing this!!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37582: [SPARK-40147][PYTHON][SQL] Make pyspark.sql.session examples self-contained

HyukjinKwon commented on code in PR #37582: URL: https://github.com/apache/spark/pull/37582#discussion_r950764394 ## python/pyspark/sql/session.py: ## @@ -99,8 +99,15 @@ def toDF(self, schema=None, sampleRatio=None): Examples ->>>

[GitHub] [spark] attilapiros commented on a diff in pull request #37474: [SPARK-40039][SS] Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

attilapiros commented on code in PR #37474: URL: https://github.com/apache/spark/pull/37474#discussion_r950764250 ## sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/CheckpointFileManagerSuite.scala: ## @@ -58,50 +77,40 @@ abstract class

[GitHub] [spark] zhengruifeng opened a new pull request, #37593: [SPARK-40111][PYTHON][DOCS] Make pyspark.rdd examples self-contained

zhengruifeng opened a new pull request, #37593: URL: https://github.com/apache/spark/pull/37593 ### What changes were proposed in this pull request? 1, Make pyspark.rdd examples self-contained 2, add `versionadded` and `set-also` ### Why are the changes needed? Make

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #37474: [SPARK-40039][SS] Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

HeartSaVioR commented on code in PR #37474: URL: https://github.com/apache/spark/pull/37474#discussion_r950751083 ## hadoop-cloud/src/hadoop-3/main/scala/org/apache/spark/internal/io/cloud/AbortableStreamBasedCheckpointFileManager.scala: ## @@ -0,0 +1,95 @@ +/* + * Licensed to

[GitHub] [spark] brandondahler commented on pull request #37361: [SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations

brandondahler commented on PR #37361: URL: https://github.com/apache/spark/pull/37361#issuecomment-1221422904 Rebased on lastest master changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] panbingkun commented on pull request #37588: [SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

panbingkun commented on PR #37588: URL: https://github.com/apache/spark/pull/37588#issuecomment-1221421651 cc @MaxGekk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] seunggabi commented on pull request #37478: [SPARK][SQL] feat: SparkSession.confing(Map)

seunggabi commented on PR #37478: URL: https://github.com/apache/spark/pull/37478#issuecomment-1221420102 @srowen You mean, A: I will make JIRA, and change title and attach link. B: you give to JIRA, and change title and attach link. A or B? I maybe think B. I

[GitHub] [spark] srowen commented on a diff in pull request #37592: [SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 2, 32 functions)

srowen commented on code in PR #37592: URL: https://github.com/apache/spark/pull/37592#discussion_r950745297 ## python/pyspark/sql/functions.py: ## @@ -994,12 +1014,22 @@ def cot(col: "ColumnOrName") -> Column: Parameters -- col :

[GitHub] [spark] srowen commented on pull request #37478: [SPARK][SQL] feat: SparkSession.confing(Map)

srowen commented on PR #37478: URL: https://github.com/apache/spark/pull/37478#issuecomment-1221412559 Please edit the title to link to the JIRA -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] khalidmammadov opened a new pull request, #37592: [SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 2, 32 functions)

khalidmammadov opened a new pull request, #37592: URL: https://github.com/apache/spark/pull/37592 ### What changes were proposed in this pull request? Docstring improvements ### Why are the changes needed? To help users to understand pyspark API ### Does this PR

[GitHub] [spark] seunggabi commented on pull request #37478: [SPARK][SQL] feat: SparkSession.confing(Map)

seunggabi commented on PR #37478: URL: https://github.com/apache/spark/pull/37478#issuecomment-1221408944 thx @srowen have a nice weekend.! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] ljfgem commented on a diff in pull request #35636: [SPARK-31357][SQL][WIP] Catalog API for view metadata

ljfgem commented on code in PR #35636: URL: https://github.com/apache/spark/pull/35636#discussion_r950735369 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -447,6 +454,74 @@ class Analyzer(override val catalogManager: CatalogManager)

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37203: [SPARK-39755][K8S] Randomization in Spark local directory for K8 resource managers

dongjoon-hyun commented on code in PR #37203: URL: https://github.com/apache/spark/pull/37203#discussion_r950734114 ## resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala: ## @@ -33,12 +34,11 @@ private[spark] class

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37203: [SPARK-39755][K8S] Randomization in Spark local directory for K8 resource managers

dongjoon-hyun commented on code in PR #37203: URL: https://github.com/apache/spark/pull/37203#discussion_r950733857 ## resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStepSuite.scala: ## @@ -52,41 +52,58 @@ class

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37203: [SPARK-39755][K8S] Randomization in Spark local directory for K8 resource managers

dongjoon-hyun commented on code in PR #37203: URL: https://github.com/apache/spark/pull/37203#discussion_r950733857 ## resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStepSuite.scala: ## @@ -52,41 +52,58 @@ class

[GitHub] [spark] MaxGekk commented on a diff in pull request #37483: [SPARK-40112][SQL] Improve the TO_BINARY() function

MaxGekk commented on code in PR #37483: URL: https://github.com/apache/spark/pull/37483#discussion_r950732296 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala: ## @@ -452,6 +452,90 @@ class StringExpressionsSuite extends

[GitHub] [spark] sunchao commented on a diff in pull request #37439: [SPARK-39896][SQL] UnwrapCastInBinaryComparison should work when the literal of In/InSet downcast failed

sunchao commented on code in PR #37439: URL: https://github.com/apache/spark/pull/37439#discussion_r950718579 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala: ## @@ -155,7 +155,16 @@ object UnwrapCastInBinaryComparison

[GitHub] [spark] zero323 commented on pull request #37329: [SPARK-39832][PYTHON] Support column arguments in regexp_replace

zero323 commented on PR #37329: URL: https://github.com/apache/spark/pull/37329#issuecomment-1221335064 Merged into master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] zero323 closed pull request #37329: [SPARK-39832][PYTHON] Support column arguments in regexp_replace

zero323 closed pull request #37329: [SPARK-39832][PYTHON] Support column arguments in regexp_replace URL: https://github.com/apache/spark/pull/37329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] srowen commented on pull request #37478: [SPARK][SQL] feat: SparkSession.confing(Map)

srowen commented on PR #37478: URL: https://github.com/apache/spark/pull/37478#issuecomment-1221325756 Last thing, I'd go ahead and make a JIRA for this. It's minor but non trivial -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] srowen commented on a diff in pull request #37591: [MINOR] Remove useless configuration & extract common code

srowen commented on code in PR #37591: URL: https://github.com/apache/spark/pull/37591#discussion_r950702204 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala: ## @@ -199,33 +199,7 @@ class ParquetFileFormat filters:

[GitHub] [spark] srowen commented on a diff in pull request #37591: [MINOR] Remove useless configuration & extract common code

srowen commented on code in PR #37591: URL: https://github.com/apache/spark/pull/37591#discussion_r950702172 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala: ## @@ -199,33 +199,7 @@ class ParquetFileFormat filters:

[GitHub] [spark] panbingkun opened a new pull request, #37591: [MINOR] Remove useless configuration & extract common code

panbingkun opened a new pull request, #37591: URL: https://github.com/apache/spark/pull/37591 ### What changes were proposed in this pull request? The pr aim to: 1. Remove useless configuration: hadoopConf.set(ParquetWriteSupport.SPARK_ROW_SCHEMA, readDataSchemaAsJson) 2. extract

[GitHub] [spark] MaxGekk opened a new pull request, #37590: [WIP][SQL] Format error messages by `spark-sql`

MaxGekk opened a new pull request, #37590: URL: https://github.com/apache/spark/pull/37590 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

[GitHub] [spark] AmplabJenkins commented on pull request #37577: [SPARK-40144][WEB UI] Standalone log-view can't load new

AmplabJenkins commented on PR #37577: URL: https://github.com/apache/spark/pull/37577#issuecomment-1221311837 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #37578: [SPARK-39791][LAUNCHER] In Spark 3.0 standalone cluster mode, unable to customize driver JVM path

AmplabJenkins commented on PR #37578: URL: https://github.com/apache/spark/pull/37578#issuecomment-1221311831 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] wangyum commented on pull request #37439: [SPARK-39896][SQL] UnwrapCastInBinaryComparison should work when the literal of In/InSet downcast failed

wangyum commented on PR #37439: URL: https://github.com/apache/spark/pull/37439#issuecomment-1221301948 cc @cloud-fan @sunchao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] wangyum opened a new pull request, #37589: [SPARK-40152][SQL] Fix split_part codegen compilation issue

wangyum opened a new pull request, #37589: URL: https://github.com/apache/spark/pull/37589 ### What changes were proposed in this pull request? Fix `split_part` codegen compilation issue: ```sql SELECT split_part(str, delimiter, partNum) FROM VALUES ('11.12.13', '.', 3) AS

[GitHub] [spark] panbingkun opened a new pull request, #37588: [SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2

panbingkun opened a new pull request, #37588: URL: https://github.com/apache/spark/pull/37588 ### What changes were proposed in this pull request? The pr aim to implement v2 SHOW TABLE EXTENDED as `ShowTableExec` ### Why are the changes needed? To have feature parity with the

[GitHub] [spark] AmplabJenkins commented on pull request #37587: [SPARK-40081][PYTHON][DOCS] Add document parameters for pyspark.sql.streaming.query

AmplabJenkins commented on PR #37587: URL: https://github.com/apache/spark/pull/37587#issuecomment-1221267996 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] dcoliversun commented on pull request #37587: [SPARK-40081][PYTHON][DOCS] Add document parameters for pyspark.sql.streaming.query

dcoliversun commented on PR #37587: URL: https://github.com/apache/spark/pull/37587#issuecomment-1221253694 cc @HyukjinKwon @xinrong-meng It would be good if you could take a look when you have time, thanks! -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] dcoliversun opened a new pull request, #37587: [SPARK-40081][PYTHON][DOCS] Add document parameters for pyspark.sql.streaming.query

dcoliversun opened a new pull request, #37587: URL: https://github.com/apache/spark/pull/37587 ### What changes were proposed in this pull request? As mentioned https://github.com/apache/spark/pull/37482#issuecomment-1212245579, this PR proposes to add document parameters

[GitHub] [spark] pralabhkumar commented on a diff in pull request #37203: [SPARK-39755][K8S] Randomization in Spark local directory for K8 resource managers

pralabhkumar commented on code in PR #37203: URL: https://github.com/apache/spark/pull/37203#discussion_r950664130 ## resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStepSuite.scala: ## @@ -52,41 +52,58 @@ class

[GitHub] [spark] pralabhkumar commented on a diff in pull request #37203: [SPARK-39755][K8S] Randomization in Spark local directory for K8 resource managers

pralabhkumar commented on code in PR #37203: URL: https://github.com/apache/spark/pull/37203#discussion_r950664881 ## resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala: ## @@ -68,9 +69,8 @@ private[spark] class

[GitHub] [spark] pralabhkumar commented on a diff in pull request #37203: [SPARK-39755][K8S] Randomization in Spark local directory for K8 resource managers

pralabhkumar commented on code in PR #37203: URL: https://github.com/apache/spark/pull/37203#discussion_r950664546 ## resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala: ## @@ -33,12 +34,11 @@ private[spark] class

[GitHub] [spark] pralabhkumar commented on a diff in pull request #37203: [SPARK-39755][K8S] Randomization in Spark local directory for K8 resource managers

pralabhkumar commented on code in PR #37203: URL: https://github.com/apache/spark/pull/37203#discussion_r950664130 ## resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStepSuite.scala: ## @@ -52,41 +52,58 @@ class

[GitHub] [spark] pan3793 commented on a diff in pull request #36995: [SPARK-39607][SQL][DSV2] Distribution and ordering support V2 function in writing

pan3793 commented on code in PR #36995: URL: https://github.com/apache/spark/pull/36995#discussion_r950662263 ## sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala: ## @@ -48,6 +48,11 @@ case class DataSourceV2Relation(

[GitHub] [spark] pan3793 commented on a diff in pull request #36995: [SPARK-39607][SQL][DSV2] Distribution and ordering support V2 function in writing

pan3793 commented on code in PR #36995: URL: https://github.com/apache/spark/pull/36995#discussion_r950662247 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala: ## @@ -143,4 +150,53 @@ object V2ExpressionUtils extends SQLConfHelper

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37203: [SPARK-39755][K8S] Randomization in Spark local directory for K8 resource managers

dongjoon-hyun commented on code in PR #37203: URL: https://github.com/apache/spark/pull/37203#discussion_r950654693 ## resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStepSuite.scala: ## @@ -52,41 +52,58 @@ class