[PR] [SPARK-47791][SQL] Truncate exceed decimals with scale first instead of precision from JDBC datasource [spark]

2024-04-09 Thread via GitHub
yaooqinn opened a new pull request, #45976: URL: https://github.com/apache/spark/pull/45976 ### What changes were proposed in this pull request? This PR is kind of a follow-up of SPARK-45905 but for JDBC datasource readings, which truncates exceed decimals with scale first

Re: [PR] [SPARK-47775][SQL] Support remaining scalar types in the variant spec. [spark]

2024-04-09 Thread via GitHub
cloud-fan commented on code in PR #45945: URL: https://github.com/apache/spark/pull/45945#discussion_r1558932654 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala: ## @@ -248,9 +253,10 @@ case object VariantGet { inp

Re: [PR] [SPARK-47591][SQL] Hive-thriftserver: Migrate logInfo with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
gengliangwang commented on code in PR #45926: URL: https://github.com/apache/spark/pull/45926#discussion_r1558922175 ## sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetColumnsOperation.scala: ## @@ -58,9 +59,12 @@ private[hive] class SparkGetC

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
cloud-fan commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558921090 ## sql/core/src/main/scala/org/apache/spark/sql/Column.scala: ## @@ -171,6 +171,26 @@ class Column(val expr: Expression) extends Logging { Column.fn(name, this,

Re: [PR] [SPARK-47591][SQL] Hive-thriftserver: Migrate logInfo with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
gengliangwang commented on code in PR #45926: URL: https://github.com/apache/spark/pull/45926#discussion_r1558920991 ## sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetCatalogsOperation.scala: ## @@ -40,7 +41,7 @@ private[hive] class SparkGetC

Re: [PR] [SPARK-47591][SQL] Hive-thriftserver: Migrate logInfo with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
gengliangwang commented on code in PR #45926: URL: https://github.com/apache/spark/pull/45926#discussion_r1558920135 ## sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala: ## @@ -126,7 +126,9 @@ private[hive] class Sp

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
cloud-fan commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558919561 ## sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/QueryContexts.scala: ## @@ -134,7 +134,9 @@ case class SQLQueryContext( override def callSite: String

Re: [PR] [SPARK-47591][SQL] Hive-thriftserver: Migrate logInfo with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
gengliangwang commented on code in PR #45926: URL: https://github.com/apache/spark/pull/45926#discussion_r1558919622 ## sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetColumnsOperation.scala: ## @@ -58,9 +59,12 @@ private[hive] class SparkGetC

Re: [PR] [SPARK-47736][SQL] Add support for AbstractArrayType [spark]

2024-04-09 Thread via GitHub
cloud-fan commented on code in PR #45891: URL: https://github.com/apache/spark/pull/45891#discussion_r1558908333 ## sql/api/src/main/scala/org/apache/spark/sql/types/ArrayType.scala: ## @@ -43,6 +43,23 @@ object ArrayType extends AbstractDataType { override private[spark] def

Re: [PR] [SPARK-47736][SQL] Add support for AbstractArrayType [spark]

2024-04-09 Thread via GitHub
cloud-fan commented on code in PR #45891: URL: https://github.com/apache/spark/pull/45891#discussion_r155890 ## sql/api/src/main/scala/org/apache/spark/sql/internal/types/AbstractArrayType.scala: ## @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] [SPARK-47792][CORE] Make the value of MDC can support `null` [spark]

2024-04-09 Thread via GitHub
panbingkun commented on PR #45975: URL: https://github.com/apache/spark/pull/45975#issuecomment-2046600800 cc @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2046598213 Thank you, @pan3793 and all. Merged to master for Apache Spark 4.0.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[PR] [SPARK-47792][CORE] Make the value of MDC can support `null` [spark]

2024-04-09 Thread via GitHub
panbingkun opened a new pull request, #45975: URL: https://github.com/apache/spark/pull/45975 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun closed pull request #45838: [SPARK-47706][BUILD] Bump json4s 4.0.7 URL: https://github.com/apache/spark/pull/45838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] [SPARK-47790][BUILD][3.5] Upgrade `commons-io` to 2.16.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun opened a new pull request, #45974: URL: https://github.com/apache/spark/pull/45974 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

Re: [PR] [SPARK-47781][SQL] Handle negative scale decimals for JDBC data sources [spark]

2024-04-09 Thread via GitHub
yaooqinn commented on PR #45956: URL: https://github.com/apache/spark/pull/45956#issuecomment-2046564369 +1, LGTM. Thank you. @cloud-fan @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] [SPARK-47781][SQL] Handle negative scale decimals for JDBC data sources [spark]

2024-04-09 Thread via GitHub
yaooqinn closed pull request #45956: [SPARK-47781][SQL] Handle negative scale decimals for JDBC data sources URL: https://github.com/apache/spark/pull/45956 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [SPARK-47790][BUILD] Upgrade `commons-io` to 2.16.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun opened a new pull request, #45973: URL: https://github.com/apache/spark/pull/45973 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

Re: [PR] [SPARK-47656][BUILD][3.5] Upgrade `commons-io` to 2.16.0 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun closed pull request #45972: [SPARK-47656][BUILD][3.5] Upgrade `commons-io` to 2.16.0 URL: https://github.com/apache/spark/pull/45972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]

2024-04-09 Thread via GitHub
pan3793 commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2046557968 @LuciferYang @dongjoon-hyun The failed test is known flaky case, I re-triggered CI, thanks for your approval and help with Mima -- This is an automated message from the Apache Git Servi

[PR] [SPARK-47656][BUILD][3.5] Upgrade `commons-io` to 2.16.0 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun opened a new pull request, #45972: URL: https://github.com/apache/spark/pull/45972 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was t

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
itholic commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558840908 ## sql/core/src/main/scala/org/apache/spark/sql/package.scala: ## @@ -111,6 +111,26 @@ package object sql { } } + private[sql] def withOrigin[T]( + pysp

Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2046536383 Could you re-trigger the failed test pipeline, @pan3793 ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] [SPARK-47593][CORE] Connector module: Migrate logWarn with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
gengliangwang closed pull request #45879: [SPARK-47593][CORE] Connector module: Migrate logWarn with variables to structured logging framework URL: https://github.com/apache/spark/pull/45879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [SPARK-47593][CORE] Connector module: Migrate logWarn with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
gengliangwang commented on PR #45879: URL: https://github.com/apache/spark/pull/45879#issuecomment-2046528714 Thanks, merging to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
itholic commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558831279 ## python/pyspark/sql/column.py: ## @@ -174,16 +175,48 @@ def _bin_op( ["Column", Union["Column", "LiteralType", "DecimalLiteral", "DateTimeLiteral"]], "Column"

Re: [PR] [SPARK-47776][SS] Disallow binary inequality collation be used in key schema of stateful operator [spark]

2024-04-09 Thread via GitHub
HeartSaVioR commented on PR #45951: URL: https://github.com/apache/spark/pull/45951#issuecomment-2046525105 Merging to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] [SPARK-47776][SS] Disallow binary inequality collation be used in key schema of stateful operator [spark]

2024-04-09 Thread via GitHub
HeartSaVioR closed pull request #45951: [SPARK-47776][SS] Disallow binary inequality collation be used in key schema of stateful operator URL: https://github.com/apache/spark/pull/45951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] [SPARK-47776][SS] Disallow binary inequality collation be used in key schema of stateful operator [spark]

2024-04-09 Thread via GitHub
HeartSaVioR commented on PR #45951: URL: https://github.com/apache/spark/pull/45951#issuecomment-2046522793 GA only failed in Docker integration which is unrelated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] [SPARK-41811][PYTHON][CONNECT] Implement `SQLStringFormatter` [spark]

2024-04-09 Thread via GitHub
zhengruifeng closed pull request #45277: [SPARK-41811][PYTHON][CONNECT] Implement `SQLStringFormatter` URL: https://github.com/apache/spark/pull/45277 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-41811][PYTHON][CONNECT] Implement `SQLStringFormatter` [spark]

2024-04-09 Thread via GitHub
zhengruifeng commented on PR #45277: URL: https://github.com/apache/spark/pull/45277#issuecomment-2046490077 close this PR in favor of https://github.com/apache/spark/pull/45614 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-47761][SQL] Oracle: Support reading AnsiIntervalTypes [spark]

2024-04-09 Thread via GitHub
yaooqinn closed pull request #45925: [SPARK-47761][SQL] Oracle: Support reading AnsiIntervalTypes URL: https://github.com/apache/spark/pull/45925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-47761][SQL] Oracle: Support reading AnsiIntervalTypes [spark]

2024-04-09 Thread via GitHub
yaooqinn commented on PR #45925: URL: https://github.com/apache/spark/pull/45925#issuecomment-2046486014 Merged to master. Thank you very much @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] [SPARK-47083][BUILD] Upgrade `commons-codec` to 1.16.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45152: URL: https://github.com/apache/spark/pull/45152#issuecomment-2046484207 Hi, All. Let me backport this to branch-3.5 to deliver the bug fixes too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] [SPARK-47787][BUILD][3.5] Upgrade `commons-compress` to 1.26.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45967: URL: https://github.com/apache/spark/pull/45967#issuecomment-2046482929 I'm checking the Scala 2.13 compilation failure currently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] [SPARK-47759][CORE] Make the critical path like RpcUtils$.askRpcTimeout less fragile [spark]

2024-04-09 Thread via GitHub
xiongbo-sjtu commented on PR #45942: URL: https://github.com/apache/spark/pull/45942#issuecomment-2046482693 > `Pattern.compile` is operating on a new instance of `Pattern` object - and so inherits the thread safety of `Pattern`. It is not clear to me what the issue being observed is, and h

Re: [PR] [SPARK-47182][BUILD] Exclude `commons-(io|lang3)` transitive dependencies from `commons-compress` and `avro*` [spark]

2024-04-09 Thread via GitHub
LuciferYang commented on PR #45278: URL: https://github.com/apache/spark/pull/45278#issuecomment-2046476828 Thanks @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] [SPARK-47182][BUILD] Exclude `commons-(io|lang3)` transitive dependencies from `commons-compress` and `avro*` [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45278: URL: https://github.com/apache/spark/pull/45278#issuecomment-2046475804 It seems that we need this transitive dependency clean up in branch-3.5 too. Let me backport this. -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] [SPARK-47767][SQL] Show offset value in TakeOrderedAndProjectExec [spark]

2024-04-09 Thread via GitHub
guixiaowen commented on code in PR #45931: URL: https://github.com/apache/spark/pull/45931#discussion_r1558776883 ## sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala: ## @@ -358,7 +358,8 @@ case class TakeOrderedAndProjectExec( val orderByString = truncate

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
itholic commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558766748 ## sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/QueryContexts.scala: ## @@ -134,7 +134,9 @@ case class SQLQueryContext( override def callSite: String =

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
itholic commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558765519 ## sql/core/src/main/scala/org/apache/spark/sql/package.scala: ## @@ -111,6 +111,26 @@ package object sql { } } + private[sql] def withOrigin[T]( + pysp

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
cloud-fan commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558765078 ## sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/QueryContexts.scala: ## @@ -134,7 +134,9 @@ case class SQLQueryContext( override def callSite: String

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
cloud-fan commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558761829 ## sql/core/src/main/scala/org/apache/spark/sql/package.scala: ## @@ -111,6 +111,26 @@ package object sql { } } + private[sql] def withOrigin[T]( + py

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
cloud-fan commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558760260 ## sql/core/src/main/scala/org/apache/spark/sql/package.scala: ## @@ -111,6 +111,26 @@ package object sql { } } + private[sql] def withOrigin[T]( + py

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
itholic commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558755161 ## sql/core/src/main/scala/org/apache/spark/sql/package.scala: ## @@ -111,6 +111,26 @@ package object sql { } } + private[sql] def withOrigin[T]( + pysp

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
itholic commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558758511 ## sql/core/src/main/scala/org/apache/spark/sql/package.scala: ## @@ -111,6 +111,26 @@ package object sql { } } + private[sql] def withOrigin[T]( + pysp

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
itholic commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558755161 ## sql/core/src/main/scala/org/apache/spark/sql/package.scala: ## @@ -111,6 +111,26 @@ package object sql { } } + private[sql] def withOrigin[T]( + pysp

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
itholic commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558755161 ## sql/core/src/main/scala/org/apache/spark/sql/package.scala: ## @@ -111,6 +111,26 @@ package object sql { } } + private[sql] def withOrigin[T]( + pysp

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
itholic commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558755161 ## sql/core/src/main/scala/org/apache/spark/sql/package.scala: ## @@ -111,6 +111,26 @@ package object sql { } } + private[sql] def withOrigin[T]( + pysp

Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]

2024-04-09 Thread via GitHub
pan3793 commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2046420830 rebased on master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [SPARK-47787][BUILD][3.5] Upgrade `commons-compress` to 1.26.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45967: URL: https://github.com/apache/spark/pull/45967#issuecomment-2046407739 Thank you, @yaooqinn ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-47781][SQL] Handle negative scale for JDBC data sources [spark]

2024-04-09 Thread via GitHub
yaooqinn commented on PR #45956: URL: https://github.com/apache/spark/pull/45956#issuecomment-2046390671 cc @dongjoon-hyun @cloud-fan, PTAL when you have some time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] [SPARK-47786] SELECT DISTINCT (*) should not become SELECT DISTINCT struct(*) (revert to previous behavior) [spark]

2024-04-09 Thread via GitHub
cloud-fan closed pull request #45970: [SPARK-47786] SELECT DISTINCT (*) should not become SELECT DISTINCT struct(*) (revert to previous behavior) URL: https://github.com/apache/spark/pull/45970 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] [SPARK-47786] SELECT DISTINCT (*) should not become SELECT DISTINCT struct(*) (revert to previous behavior) [spark]

2024-04-09 Thread via GitHub
cloud-fan commented on PR #45970: URL: https://github.com/apache/spark/pull/45970#issuecomment-2046357960 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [WIP][SPARK-41811][PYTHON][CONNECT] Implement `SQLStringFormatter` with `WithRelations` [spark]

2024-04-09 Thread via GitHub
zhengruifeng commented on code in PR #45614: URL: https://github.com/apache/spark/pull/45614#discussion_r1558712909 ## python/pyspark/sql/session.py: ## @@ -1662,7 +1662,7 @@ def sql( And substitute named parameters with the `:` prefix by SQL literals. ->>>

Re: [PR] [SPARK-47781][SQL] Handle negative scale and truncate exceed scale first for JDBC data sources [spark]

2024-04-09 Thread via GitHub
yaooqinn commented on code in PR #45956: URL: https://github.com/apache/spark/pull/45956#discussion_r1558703743 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala: ## @@ -196,9 +196,13 @@ object JdbcUtils extends Logging with SQLConfHelper

Re: [PR] [SPARK-47586][SQL] Hive module: Migrate logError with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
itholic commented on PR #45876: URL: https://github.com/apache/spark/pull/45876#issuecomment-2046329098 Thanks @gengliangwang @panbingkun @pan3793 for the review :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [SPARK-47586][SQL] Hive module: Migrate logError with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
gengliangwang closed pull request #45876: [SPARK-47586][SQL] Hive module: Migrate logError with variables to structured logging framework URL: https://github.com/apache/spark/pull/45876 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] [SPARK-47586][SQL] Hive module: Migrate logError with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
gengliangwang commented on PR #45876: URL: https://github.com/apache/spark/pull/45876#issuecomment-2046286778 Thanks, merging to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [SPARK-47784][SS] Merge TTLMode and TimeoutMode into a single TimeMode. [spark]

2024-04-09 Thread via GitHub
sahnib commented on code in PR #45960: URL: https://github.com/apache/spark/pull/45960#discussion_r1558595818 ## common/utils/src/main/resources/error/error-classes.json: ## @@ -3603,6 +3597,12 @@ ], "sqlState" : "42802" }, + "STATEFUL_PROCESSOR_INCORRECT_TIME_MODE

Re: [PR] [SPARK-47673][SS] Implementing TTL for ListState [spark]

2024-04-09 Thread via GitHub
anishshri-db commented on code in PR #45932: URL: https://github.com/apache/spark/pull/45932#discussion_r1558571847 ## sql/api/src/main/scala/org/apache/spark/sql/streaming/StatefulProcessorHandle.scala: ## @@ -72,6 +72,26 @@ private[sql] trait StatefulProcessorHandle extends S

Re: [PR] [SPARK-47673][SS] Implementing TTL for ListState [spark]

2024-04-09 Thread via GitHub
anishshri-db commented on code in PR #45932: URL: https://github.com/apache/spark/pull/45932#discussion_r1558570672 ## sql/api/src/main/scala/org/apache/spark/sql/streaming/ListState.scala: ## @@ -44,3 +44,23 @@ private[sql] trait ListState[S] extends Serializable { /** Remov

Re: [PR] [SPARK-47673][SS] Implementing TTL for ListState [spark]

2024-04-09 Thread via GitHub
anishshri-db commented on PR #45932: URL: https://github.com/apache/spark/pull/45932#issuecomment-2046265049 Should we update the PR description ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-47784][SS] Merge TTLMode and TimeoutMode into a single TimeMode. [spark]

2024-04-09 Thread via GitHub
anishshri-db commented on code in PR #45960: URL: https://github.com/apache/spark/pull/45960#discussion_r1558549967 ## docs/sql-error-conditions.md: ## @@ -2183,30 +2183,30 @@ The SQL config `` cannot be found. Please verify that the config exists Star (*) is not allowed in

Re: [PR] [SPARK-47787][BUILD] Upgrade `commons-compress` to 1.26.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun closed pull request #45966: [SPARK-47787][BUILD] Upgrade `commons-compress` to 1.26.1 URL: https://github.com/apache/spark/pull/45966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] [SPARK-47787][BUILD] Upgrade `commons-compress` to 1.26.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45966: URL: https://github.com/apache/spark/pull/45966#issuecomment-2046254867 Thank you! Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [DOCS][MINOR] Fix typos [spark]

2024-04-09 Thread via GitHub
github-actions[bot] closed pull request #44276: [DOCS][MINOR] Fix typos URL: https://github.com/apache/spark/pull/44276 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

[PR] [SPARK-47786] SELECT DISTINCT (*) should not become SELECT DISTINCT struct(*) (revert to previous behavior) [spark]

2024-04-09 Thread via GitHub
srielau opened a new pull request, #45970: URL: https://github.com/apache/spark/pull/45970 ### What changes were proposed in this pull request? We special case SELECT DISTINCT (*) to become SELECT DISTINCT * This prevents (*) to be treated as struct(*). We used to ignore

Re: [PR] [SPARK-47777][PYTHON][SS][TESTS] Add spark connect test for python streaming data source [spark]

2024-04-09 Thread via GitHub
chaoqin-li1123 commented on PR #45950: URL: https://github.com/apache/spark/pull/45950#issuecomment-2046225155 cc @allisonwang-db @HeartSaVioR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-47581][CORE] SQL core: Migrate logError with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
dtenedor commented on PR #45969: URL: https://github.com/apache/spark/pull/45969#issuecomment-2046193017 cc @gengliangwang here is the structured logging migration for `logError` within sql/core. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] [SPARK-47787][BUILD] Upgrade `commons-compress` to 1.26.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45966: URL: https://github.com/apache/spark/pull/45966#issuecomment-2046192086 Could you review this PR, @viirya ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[PR] [SPARK-47581][CORE] SQL core: Migrate logError with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
dtenedor opened a new pull request, #45969: URL: https://github.com/apache/spark/pull/45969 ### What changes were proposed in this pull request? Migrate logError with variables of the sql/core module to structured logging framework. This transforms the logError entries of the followin

Re: [PR] [SPARK-47318][CORE] Adds HKDF round to AuthEngine key derivation to follow standard KEX practices [spark]

2024-04-09 Thread via GitHub
sweisdb commented on PR #45425: URL: https://github.com/apache/spark/pull/45425#issuecomment-2046170137 > Ya, please use `dev/lint-java` script and fix the following lines, @sweisdb . > > ``` > $ dev/lint-java > Using `mvn` from path: /opt/homebrew/bin/mvn > Using SPARK_LOCAL

Re: [PR] [SPARK-47733][SS] Add custom metrics for transformWithState operator part of query progress [spark]

2024-04-09 Thread via GitHub
anishshri-db commented on PR #45937: URL: https://github.com/apache/spark/pull/45937#issuecomment-2046169178 @sahnib @HeartSaVioR - PTAL, thx ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] Ms ttl [spark]

2024-04-09 Thread via GitHub
ericm-db opened a new pull request, #45968: URL: https://github.com/apache/spark/pull/45968 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How wa

[PR] [SPARK-47787][BUILD][3.5] Upgrade `commons-compress` to 1.26.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun opened a new pull request, #45967: URL: https://github.com/apache/spark/pull/45967 ### What changes were proposed in this pull request? This PR aims to upgrade `commons-compress` to 1.26.1. ### Why are the changes needed? To bring the latest bug fixes. -

[PR] [SPARK-47787][BUILD] Upgrade `commons-compress` to 1.26.1 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun opened a new pull request, #45966: URL: https://github.com/apache/spark/pull/45966 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

Re: [PR] [SPARK-47109][BUILD][3.4] Upgrade `commons-compress` to 1.26.0 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun closed pull request #45965: [SPARK-47109][BUILD][3.4] Upgrade `commons-compress` to 1.26.0 URL: https://github.com/apache/spark/pull/45965 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-47109][BUILD][3.5] Upgrade `commons-compress` to 1.26.0 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun closed pull request #45964: [SPARK-47109][BUILD][3.5] Upgrade `commons-compress` to 1.26.0 URL: https://github.com/apache/spark/pull/45964 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] [SPARK-47109][BUILD][3.4] Upgrade `commons-compress` to 1.26.0 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun opened a new pull request, #45965: URL: https://github.com/apache/spark/pull/45965 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

[PR] [SPARK-47109][BUILD][3.5] Upgrade `commons-compress` to 1.26.0 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun opened a new pull request, #45964: URL: https://github.com/apache/spark/pull/45964 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

Re: [PR] [DO-NOT-REVEW][SPARK-47718] temp view loses wm [spark]

2024-04-09 Thread via GitHub
WweiL commented on PR #45888: URL: https://github.com/apache/spark/pull/45888#issuecomment-2046014994 closing this, this is a customer error -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] [DO-NOT-REVEW][SPARK-47718] temp view loses wm [spark]

2024-04-09 Thread via GitHub
WweiL closed pull request #45888: [DO-NOT-REVEW][SPARK-47718] temp view loses wm URL: https://github.com/apache/spark/pull/45888 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-47785][BUILD][TESTS] Upgrade `bouncycastle` to 1.78 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45962: URL: https://github.com/apache/spark/pull/45962#issuecomment-2045947467 Thank you, @huaxingao ! Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] [SPARK-47785][BUILD][TESTS] Upgrade `bouncycastle` to 1.78 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun closed pull request #45962: [SPARK-47785][BUILD][TESTS] Upgrade `bouncycastle` to 1.78 URL: https://github.com/apache/spark/pull/45962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [SPARK-47783] Add some missing SQLSTATEs an clean up the YY000 to use… [spark]

2024-04-09 Thread via GitHub
gengliangwang closed pull request #45961: [SPARK-47783] Add some missing SQLSTATEs an clean up the YY000 to use… URL: https://github.com/apache/spark/pull/45961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] [SPARK-47783] Add some missing SQLSTATEs an clean up the YY000 to use… [spark]

2024-04-09 Thread via GitHub
gengliangwang commented on PR #45961: URL: https://github.com/apache/spark/pull/45961#issuecomment-2045863691 Thanks, merging to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [WIP][SPARK-41811][PYTHON][CONNECT] Implement `SQLStringFormatter` with `WithRelations` [spark]

2024-04-09 Thread via GitHub
xinrong-meng commented on PR #45614: URL: https://github.com/apache/spark/pull/45614#issuecomment-2045857867 LGTM after resolving conflicts, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] [SPARK-47785][BUILD][TESTS] Upgrade `bouncycastle` to 1.78 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45962: URL: https://github.com/apache/spark/pull/45962#issuecomment-2045855121 Could you review this test dependency PR, @huaxingao ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] [SPARK-47785][BUILD][TESTS] Upgrade `bouncycastle` to 1.78 [spark]

2024-04-09 Thread via GitHub
dongjoon-hyun commented on PR #45962: URL: https://github.com/apache/spark/pull/45962#issuecomment-2045852676 All tests passed except the known CSV failures which were fixed already in `master` branch an hour ago. - https://github.com/apache/spark/pull/45958 -- This is an automated mes

Re: [PR] [WIP][SPARK-41811][PYTHON][CONNECT] Implement `SQLStringFormatter` with `WithRelations` [spark]

2024-04-09 Thread via GitHub
xinrong-meng commented on code in PR #45614: URL: https://github.com/apache/spark/pull/45614#discussion_r1558131447 ## python/pyspark/sql/session.py: ## @@ -1662,7 +1662,7 @@ def sql( And substitute named parameters with the `:` prefix by SQL literals. ->>>

Re: [PR] [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors [spark]

2024-04-09 Thread via GitHub
xinrong-meng commented on code in PR #45377: URL: https://github.com/apache/spark/pull/45377#discussion_r1558128891 ## python/pyspark/errors/exceptions/captured.py: ## @@ -379,5 +379,13 @@ def fragment(self) -> str: def callSite(self) -> str: return str(self._q.cal

Re: [PR] [SPARK-47777][PYTHON][SS][TESTS] Add spark connect test for python streaming data source [spark]

2024-04-09 Thread via GitHub
xinrong-meng commented on PR #45950: URL: https://github.com/apache/spark/pull/45950#issuecomment-2045843912 LGTM once CI pass, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [SPARK-47775][SQL] Support remaining scalar types in the variant spec. [spark]

2024-04-09 Thread via GitHub
chenhao-db commented on code in PR #45945: URL: https://github.com/apache/spark/pull/45945#discussion_r1558098749 ## common/variant/src/main/java/org/apache/spark/types/variant/Variant.java: ## @@ -89,6 +97,37 @@ public BigDecimal getDecimal() { return VariantUtil.getDecima

[PR] [SPARK-47415][SQL] Collation support: Levenshtein [spark]

2024-04-09 Thread via GitHub
nikolamand-db opened a new pull request, #45963: URL: https://github.com/apache/spark/pull/45963 ### What changes were proposed in this pull request? Levenshtein distance requires modifications to accommodate collated inputs properly: - Allow collated inputs in `inputTy

Re: [PR] [SPARK-47777][PYTHON][SS][TESTS] Add spark connect test for python streaming data source [spark]

2024-04-09 Thread via GitHub
chaoqin-li1123 commented on code in PR #45950: URL: https://github.com/apache/spark/pull/45950#discussion_r1558088128 ## python/pyspark/sql/tests/connect/test_parity_python_streaming_datasource.py: ## @@ -0,0 +1,35 @@ +# Review Comment: Added. -- This is an automated mes

Re: [PR] [SPARK-47581][CORE][FOLLOWUP] Fix GA failure [spark]

2024-04-09 Thread via GitHub
dtenedor commented on PR #45958: URL: https://github.com/apache/spark/pull/45958#issuecomment-2045763707 Thanks @gengliangwang @panbingkun for the fix!! And apologies for the breakage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [SPARK-47581][CORE] SQL catalyst: Migrate logWarning with variables to structured logging framework [spark]

2024-04-09 Thread via GitHub
dtenedor commented on PR #45904: URL: https://github.com/apache/spark/pull/45904#issuecomment-2045764245 Thanks @panbingkun for the fix!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-47545][CONNECT] Dataset `observe` support for the Scala client [spark]

2024-04-09 Thread via GitHub
hvanhovell commented on code in PR #45701: URL: https://github.com/apache/spark/pull/45701#discussion_r1558066618 ## connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/CheckConnectJvmClientCompatibility.scala: ## @@ -363,6 +363,8 @@ object CheckConne

Re: [PR] [SPARK-47736][SQL] Add support for AbstractArrayType [spark]

2024-04-09 Thread via GitHub
mihailom-db commented on PR #45891: URL: https://github.com/apache/spark/pull/45891#issuecomment-2045750423 @cloud-fan Did some refactoring with some renaming, so I re-requested approve -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

  1   2   >