Re: [PR] [SPARK-48985][CONNECT] Connect Compatible Expression Constructors [spark]

2024-07-27 Thread via GitHub
asfgit closed pull request #47464: [SPARK-48985][CONNECT] Connect Compatible Expression Constructors URL: https://github.com/apache/spark/pull/47464 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [SPARK-48985][CONNECT] Connect Compatible Expression Constructors [spark]

2024-07-27 Thread via GitHub
hvanhovell commented on PR #47464: URL: https://github.com/apache/spark/pull/47464#issuecomment-2254315594 Merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] [SPARK-49031] Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2 [spark]

2024-07-27 Thread via GitHub
ericm-db commented on code in PR #47508: URL: https://github.com/apache/spark/pull/47508#discussion_r1694043815 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/OperatorStateMetadata.scala: ## @@ -89,6 +91,40 @@ case class OperatorStateMetadataV2( ove

Re: [PR] [SPARK-49031] Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2 [spark]

2024-07-27 Thread via GitHub
anishshri-db commented on code in PR #47508: URL: https://github.com/apache/spark/pull/47508#discussion_r1694039021 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/OperatorStateMetadata.scala: ## @@ -89,6 +91,40 @@ case class OperatorStateMetadataV2(

Re: [PR] [SPARK-49031] Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2 [spark]

2024-07-27 Thread via GitHub
anishshri-db commented on code in PR #47508: URL: https://github.com/apache/spark/pull/47508#discussion_r1694037393 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateVariableUtils.scala: ## @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Softw

Re: [PR] [SPARK-49031] Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2 [spark]

2024-07-27 Thread via GitHub
anishshri-db commented on code in PR #47508: URL: https://github.com/apache/spark/pull/47508#discussion_r1694037223 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala: ## @@ -667,3 +711,32 @@ object TransformWithStateExec { } // sc

Re: [PR] [SPARK-47870][SQL] Optimize predicate after push extra predicate through join [spark]

2024-07-27 Thread via GitHub
github-actions[bot] closed pull request #46085: [SPARK-47870][SQL] Optimize predicate after push extra predicate through join URL: https://github.com/apache/spark/pull/46085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [SPARK-47336][SQL][CONNECT] Provide to PySpark a functionality to get estimated size of DataFrame in bytes [spark]

2024-07-27 Thread via GitHub
SemyonSinchenko commented on PR #46368: URL: https://github.com/apache/spark/pull/46368#issuecomment-2254250916 @HyukjinKwon @zhengruifeng Sorry for tagging but maybe you can take a look again? I fixed everything from the last review round... Tnx in advance! -- This is an automated messag

Re: [PR] [SPARK-48700] [SQL] Mode expression for complex types (all collations) [spark]

2024-07-27 Thread via GitHub
uros-db commented on code in PR #47154: URL: https://github.com/apache/spark/pull/47154#discussion_r1694013340 ## sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala: ## @@ -1789,44 +1798,90 @@ class CollationSQLExpressionsSuite s"named_str

Re: [PR] [SPARK-48700] [SQL] Mode expression for complex types (all collations) [spark]

2024-07-27 Thread via GitHub
uros-db commented on code in PR #47154: URL: https://github.com/apache/spark/pull/47154#discussion_r1694012773 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Mode.scala: ## @@ -106,11 +155,13 @@ case class Mode( val collationAwareBuffer =

Re: [PR] [SPARK-48936][CONNECT] Makes spark-shell works with Spark connect [spark]

2024-07-27 Thread via GitHub
HyukjinKwon commented on PR #47402: URL: https://github.com/apache/spark/pull/47402#issuecomment-2254243027 Vote passed. This should be ready for a look, cc @hvanhovell -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [SPARK-49013] Change key in collationsMap for Map and Array types in scala [spark]

2024-07-27 Thread via GitHub
HyukjinKwon closed pull request #47497: [SPARK-49013] Change key in collationsMap for Map and Array types in scala URL: https://github.com/apache/spark/pull/47497 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] [SPARK-49013] Change key in collationsMap for Map and Array types in scala [spark]

2024-07-27 Thread via GitHub
HyukjinKwon commented on PR #47497: URL: https://github.com/apache/spark/pull/47497#issuecomment-2254241810 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-49008][PYTHON] Use `ParamSpec` to propagate `func` signature in `transform` [spark]

2024-07-27 Thread via GitHub
HyukjinKwon commented on code in PR #47493: URL: https://github.com/apache/spark/pull/47493#discussion_r1694011602 ## python/pyspark/sql/dataframe.py: ## @@ -30,6 +30,7 @@ overload, TYPE_CHECKING, ) +from typing_extensions import Concatenate, ParamSpec Review Comment

Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]

2024-07-27 Thread via GitHub
HyukjinKwon commented on PR #47500: URL: https://github.com/apache/spark/pull/47500#issuecomment-2254241177 hm, this works to be with this change: ```bash sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false starting org.apache.spark.sql.connect.service.S

Re: [PR] [SPARK-49016][SQL] Queries from raw CSV files are disallowed when the referenced columns only include the internal corrupt record column [spark]

2024-07-27 Thread via GitHub
HyukjinKwon commented on code in PR #47506: URL: https://github.com/apache/spark/pull/47506#discussion_r1694010916 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala: ## @@ -1739,6 +1739,32 @@ abstract class CSVSuite Row(1, Date.value

Re: [PR] [SPARK-49016][SQL] Queries from raw CSV files are disallowed when the referenced columns only include the internal corrupt record column [spark]

2024-07-27 Thread via GitHub
HyukjinKwon commented on code in PR #47506: URL: https://github.com/apache/spark/pull/47506#discussion_r1694010907 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala: ## @@ -1739,6 +1739,32 @@ abstract class CSVSuite Row(1, Date.value

Re: [PR] [SPARK-48700] [SQL] Mode expression for complex types (all collations) [spark]

2024-07-27 Thread via GitHub
GideonPotok commented on code in PR #47154: URL: https://github.com/apache/spark/pull/47154#discussion_r1693998767 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Mode.scala: ## @@ -106,11 +155,13 @@ case class Mode( val collationAwareBuff

Re: [PR] [SPARK-48985][CONNECT] Connect Compatible Expression Constructors [spark]

2024-07-27 Thread via GitHub
hvanhovell commented on code in PR #47464: URL: https://github.com/apache/spark/pull/47464#discussion_r1693954290 ## connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -1893,33 +1855,6 @@ class SparkConnectPlanner( val un

Re: [PR] [SPARK-42307][SQL] Assign name for error _LEGACY_ERROR_TEMP_2232 [spark]

2024-07-27 Thread via GitHub
junyuc25 commented on code in PR #47354: URL: https://github.com/apache/spark/pull/47354#discussion_r1693947793 ## sql/core/src/test/scala/org/apache/spark/sql/RowSuite.scala: ## @@ -123,4 +123,17 @@ class RowSuite extends SparkFunSuite with SharedSparkSession { paramete

Re: [PR] [SPARK-49003][SQL] Ilicmarkodb/fix string hash [spark]

2024-07-27 Thread via GitHub
ilicmarkodb commented on code in PR #47502: URL: https://github.com/apache/spark/pull/47502#discussion_r1693311745 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala: ## @@ -620,6 +620,33 @@ class HashExpressionsSuite extends Spar

Re: [PR] [SPARK-49034][CORE] Support server-side `sparkProperties` replacement in REST Submission API [spark]

2024-07-27 Thread via GitHub
dongjoon-hyun commented on PR #47511: URL: https://github.com/apache/spark/pull/47511#issuecomment-2253983087 Merged to master for Apache Spark 4.0.0-preview2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] [SPARK-49034][CORE] Support server-side `sparkProperties` replacement in REST Submission API [spark]

2024-07-27 Thread via GitHub
dongjoon-hyun closed pull request #47511: [SPARK-49034][CORE] Support server-side `sparkProperties` replacement in REST Submission API URL: https://github.com/apache/spark/pull/47511 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] [SPARK-49034][CORE] Support server-side `sparkProperties` replacement in REST Submission API [spark]

2024-07-27 Thread via GitHub
dongjoon-hyun commented on PR #47511: URL: https://github.com/apache/spark/pull/47511#issuecomment-2253932363 Thank you, @yaooqinn . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-49002][SQL] Consistently handle invalid location/path values for all database objects [spark]

2024-07-27 Thread via GitHub
yaooqinn commented on PR #47485: URL: https://github.com/apache/spark/pull/47485#issuecomment-2253877409 Thank you @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] [SPARK-49033][CORE] Support server-side `environmentVariables` replacement in REST Submission API [spark]

2024-07-27 Thread via GitHub
yaooqinn commented on PR #47509: URL: https://github.com/apache/spark/pull/47509#issuecomment-2253875178 @dongjoon-hyun, Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co