mihailom-db commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547242430
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCasts.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundat
zhengruifeng commented on code in PR #45796:
URL: https://github.com/apache/spark/pull/45796#discussion_r1547242375
##
sql/core/src/main/scala/org/apache/spark/sql/Column.scala:
##
@@ -1222,6 +1222,43 @@ class Column(val expr: Expression) extends Logging {
*/
def cast(to:
cloud-fan commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547239943
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
##
@@ -702,26 +707,39 @@ abstract class TypeCoercionBase {
}.getOrE
zhengruifeng commented on code in PR #45796:
URL: https://github.com/apache/spark/pull/45796#discussion_r1547238526
##
connector/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##
@@ -137,6 +137,13 @@ message Expression {
}
}
+ enum EvalMode {
+E
mihailom-db commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547226617
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
##
@@ -702,26 +707,39 @@ abstract class TypeCoercionBase {
}.getO
dongjoon-hyun commented on PR #45812:
URL: https://github.com/apache/spark/pull/45812#issuecomment-2031197623
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific commen
dongjoon-hyun closed pull request #45812: [SPARK-47686][SQL][TESTS] Use `=!=`
instead of `!==` in `JoinHintSuite`
URL: https://github.com/apache/spark/pull/45812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abo
mihailom-db commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547226867
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
##
@@ -702,26 +707,39 @@ abstract class TypeCoercionBase {
}.getO
dongjoon-hyun commented on PR #45812:
URL: https://github.com/apache/spark/pull/45812#issuecomment-2031196546
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
cloud-fan commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547210423
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCasts.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundatio
mihailom-db commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547201438
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCasts.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundat
anishshri-db commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547200600
##
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatefulProcessorHandleSuite.scala:
##
@@ -48,7 +49,7 @@ class StatefulProcessorHandleSui
anishshri-db commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547198717
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ValueStateImplWithTTL.scala:
##
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Found
anishshri-db commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547199470
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ValueStateImplWithTTL.scala:
##
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Found
anishshri-db commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547198338
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ValueStateImplWithTTL.scala:
##
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Found
cloud-fan commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547197835
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:
##
@@ -702,26 +707,39 @@ abstract class TypeCoercionBase {
}.getOrE
cloud-fan commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547196013
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCasts.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundatio
anishshri-db commented on code in PR #45778:
URL: https://github.com/apache/spark/pull/45778#discussion_r1547194931
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:
##
@@ -206,19 +208,24 @@ class PrefixKeyScanStateEncoder(
*
cloud-fan commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547193589
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCasts.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundatio
anishshri-db commented on code in PR #45778:
URL: https://github.com/apache/spark/pull/45778#discussion_r1547191718
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:
##
@@ -276,53 +283,113 @@ class RangeKeyScanStateEncoder(
cloud-fan commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547191461
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCasts.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundatio
LuciferYang commented on code in PR #45812:
URL: https://github.com/apache/spark/pull/45812#discussion_r1547191432
##
sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala:
##
@@ -695,11 +695,11 @@ class JoinHintSuite extends PlanTest with
SharedSparkSession with Ada
LuciferYang commented on code in PR #45812:
URL: https://github.com/apache/spark/pull/45812#discussion_r1547190468
##
sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala:
##
@@ -695,11 +695,11 @@ class JoinHintSuite extends PlanTest with
SharedSparkSession with Ada
anishshri-db commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547189404
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ValueStateImpl.scala:
##
@@ -16,39 +16,45 @@
*/
package org.apache.spark.sql.execution.strea
cloud-fan commented on code in PR #45383:
URL: https://github.com/apache/spark/pull/45383#discussion_r1547189187
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCasts.scala:
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundatio
sahnib commented on code in PR #45778:
URL: https://github.com/apache/spark/pull/45778#discussion_r1547188291
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:
##
@@ -276,53 +283,113 @@ class RangeKeyScanStateEncoder(
range
anishshri-db commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547187515
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TTLState.scala:
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) u
anishshri-db commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547187515
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TTLState.scala:
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) u
dongjoon-hyun commented on code in PR #45812:
URL: https://github.com/apache/spark/pull/45812#discussion_r1547187101
##
sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala:
##
@@ -695,11 +695,11 @@ class JoinHintSuite extends PlanTest with
SharedSparkSession with A
dongjoon-hyun commented on code in PR #45812:
URL: https://github.com/apache/spark/pull/45812#discussion_r1547186743
##
sql/core/src/test/scala/org/apache/spark/sql/JoinHintSuite.scala:
##
@@ -695,11 +695,11 @@ class JoinHintSuite extends PlanTest with
SharedSparkSession with A
anishshri-db commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547185002
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImpl.scala:
##
@@ -77,14 +78,23 @@ class StatefulProcessorHandleImpl(
panbingkun commented on PR #45808:
URL: https://github.com/apache/spark/pull/45808#issuecomment-2031144226
> @panbingkun Thanks for the work. LGTM except some minor comments.
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
anishshri-db commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547183958
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -49,12 +54,17 @@ object StateKeyValueRowSchema {
class Sta
dongjoon-hyun commented on PR #45801:
URL: https://github.com/apache/spark/pull/45801#issuecomment-2031142759
Sorry guys, @HyukjinKwon and @attilapiros .
It seems to require more efforts than I thought.
I'll revisit this later in this week.
--
This is an automated message from the A
LuciferYang opened a new pull request, #45812:
URL: https://github.com/apache/spark/pull/45812
### What changes were proposed in this pull request?
This pr use `=!=` instead of `!==` in `JoinHintSuite`. `!==` is a
deprecated API since 2.0.0, and its test already exists in `DeprecatedAPIS
anishshri-db commented on code in PR #45778:
URL: https://github.com/apache/spark/pull/45778#discussion_r1547174481
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:
##
@@ -283,45 +286,79 @@ class RangeKeyScanStateEncoder(
//
LuciferYang commented on PR #45811:
URL: https://github.com/apache/spark/pull/45811#issuecomment-2031125874
cc @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment
LuciferYang commented on code in PR #45811:
URL: https://github.com/apache/spark/pull/45811#discussion_r1547170083
##
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##
@@ -1666,11 +1666,17 @@ class DataFrameAggregateSuite extends QueryTest
asser
yaooqinn commented on code in PR #45810:
URL: https://github.com/apache/spark/pull/45810#discussion_r1547169844
##
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala:
##
@@ -389,24 +390,25 @@ class PostgresIntegrationSuite
LuciferYang commented on code in PR #45811:
URL: https://github.com/apache/spark/pull/45811#discussion_r1547169228
##
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##
@@ -1666,11 +1666,17 @@ class DataFrameAggregateSuite extends QueryTest
asser
LuciferYang commented on code in PR #45811:
URL: https://github.com/apache/spark/pull/45811#discussion_r1547169054
##
sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala:
##
@@ -58,10 +58,12 @@ class RelationalGroupedDataset protected[sql](
import Rela
LuciferYang opened a new pull request, #45811:
URL: https://github.com/apache/spark/pull/45811
### What changes were proposed in this pull request?
When I reviewed the changes in SPARK-45685, I found an old user case that is
no longer supported:
```scala
Seq(1).toDF("id").groupB
yaooqinn commented on code in PR #45810:
URL: https://github.com/apache/spark/pull/45810#discussion_r1547165738
##
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala:
##
@@ -389,24 +390,25 @@ class PostgresIntegrationSuite
panbingkun commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547159013
##
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala:
##
@@ -33,7 +33,7 @@ import org.apache.spark.
panbingkun commented on PR #45808:
URL: https://github.com/apache/spark/pull/45808#issuecomment-203986
I will update `soon` so that we can proceed smoothly with the following work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
panbingkun commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547158917
##
common/utils/src/test/scala/org/apache/spark/util/MDCSuite.scala:
##
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+
panbingkun commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547158697
##
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##
@@ -21,5 +21,9 @@ package org.apache.spark.internal
* All structured logging keys should
panbingkun commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547156682
##
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##
@@ -21,5 +21,9 @@ package org.apache.spark.internal
* All structured logging keys should
panbingkun commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547156577
##
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##
@@ -57,12 +58,14 @@ trait LoggingSuiteBase
def msgWithMDCAndException: Log
LuciferYang commented on PR #45800:
URL: https://github.com/apache/spark/pull/45800#issuecomment-2031106167
late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To uns
dongjoon-hyun commented on PR #45801:
URL: https://github.com/apache/spark/pull/45801#issuecomment-2031104363
Thank you, @attilapiros . I addressed your comment and fixed the patch
according to HIVE-21078 and HIVE-21164.
It seems that there are more API changes from Hive side. I'm still l
dongjoon-hyun commented on code in PR #45810:
URL: https://github.com/apache/spark/pull/45810#discussion_r1547154074
##
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala:
##
@@ -389,24 +390,25 @@ class PostgresIntegrationS
yaooqinn commented on code in PR #45810:
URL: https://github.com/apache/spark/pull/45810#discussion_r1547147771
##
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala:
##
@@ -389,24 +390,25 @@ class PostgresIntegrationSuite
dongjoon-hyun commented on code in PR #45810:
URL: https://github.com/apache/spark/pull/45810#discussion_r1547144486
##
connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala:
##
@@ -389,24 +390,25 @@ class PostgresIntegrationS
ueshin commented on code in PR #45796:
URL: https://github.com/apache/spark/pull/45796#discussion_r1547142056
##
connector/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##
@@ -137,6 +137,13 @@ message Expression {
}
}
+ enum EvalMode {
+EVAL_MO
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547143112
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TTLState.scala:
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under o
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547142763
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImpl.scala:
##
@@ -185,6 +202,16 @@ class StatefulProcessorHandleImpl(
ti
yaooqinn commented on PR #45789:
URL: https://github.com/apache/spark/pull/45789#issuecomment-2031086285
Thank you @dongjoon-hyun. Merged to master.
PS: The impact on existing workloads is trivial, as users are still able to
write shorts to pre-defined int-tables
--
This is an
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547142418
##
sql/api/src/main/scala/org/apache/spark/sql/streaming/ValueState.scala:
##
@@ -42,8 +43,22 @@ private[sql] trait ValueState[S] extends Serializable {
/** Get the s
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547141805
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TTLState.scala:
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under o
yaooqinn closed pull request #45789: [SPARK-47665][SQL] Use SMALLINT to Write
ShortType to MYSQL
URL: https://github.com/apache/spark/pull/45789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the s
yaooqinn opened a new pull request, #45810:
URL: https://github.com/apache/spark/pull/45810
### What changes were proposed in this pull request?
This PR maps length unspecified bpchar to StringType for Postgres. Length
unspecified bpchar represents variable unlimited chara
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547140572
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TTLState.scala:
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under o
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547140114
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImpl.scala:
##
@@ -77,14 +78,23 @@ class StatefulProcessorHandleImpl(
sto
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547139448
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImpl.scala:
##
@@ -103,22 +113,29 @@ class StatefulProcessorHandleImpl(
p
gengliangwang commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547137886
##
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##
@@ -57,12 +58,14 @@ trait LoggingSuiteBase
def msgWithMDCAndException:
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547138671
##
common/utils/src/main/resources/error/error-classes.json:
##
@@ -3565,6 +3571,19 @@
],
"sqlState" : "42802"
},
+ "STATEFUL_PROCESSOR_CANNOT_USE_TTL_DURA
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547138251
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ValueStateImplWithTTL.scala:
##
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation
gengliangwang commented on PR #45808:
URL: https://github.com/apache/spark/pull/45808#issuecomment-2031080357
@panbingkun Thanks for the work. LGTM except some minor comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub an
gengliangwang commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547135454
##
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala:
##
@@ -33,7 +33,7 @@ import org.apache.spa
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547135608
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ValueStateImpl.scala:
##
@@ -16,39 +16,46 @@
*/
package org.apache.spark.sql.execution.streaming
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547135223
##
sql/api/src/main/java/org/apache/spark/sql/streaming/TTLMode.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547134400
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -49,12 +54,17 @@ object StateKeyValueRowSchema {
class StateType
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547134168
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -49,12 +54,17 @@ object StateKeyValueRowSchema {
class StateType
sahnib commented on code in PR #45674:
URL: https://github.com/apache/spark/pull/45674#discussion_r1547132684
##
sql/api/src/main/scala/org/apache/spark/sql/streaming/ValueState.scala:
##
@@ -42,8 +43,22 @@ private[sql] trait ValueState[S] extends Serializable {
/** Get the s
gengliangwang commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547130360
##
common/utils/src/test/scala/org/apache/spark/util/MDCSuite.scala:
##
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or mor
gengliangwang commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547129698
##
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##
@@ -21,5 +21,9 @@ package org.apache.spark.internal
* All structured logging keys shou
dongjoon-hyun closed pull request #45804: [SPARK-47679][SQL] Use
`HiveConf.getConfVars` or Hive conf names directly
URL: https://github.com/apache/spark/pull/45804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL a
dongjoon-hyun commented on PR #45804:
URL: https://github.com/apache/spark/pull/45804#issuecomment-2031042389
Merged to master for Apache Spark 4.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
leixm commented on PR #45765:
URL: https://github.com/apache/spark/pull/45765#issuecomment-2031020720
Benchmark result:
```
[info] JSON functions: Best Time(ms) Avg
Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative
[info]
---
leixm commented on PR #45765:
URL: https://github.com/apache/spark/pull/45765#issuecomment-2031020085
I deleted below code, this ut will cause an error(Assignment conversion not
possible from type "scala.collection.IterableOnce" to type
"org.apache.spark.sql.catalyst.util.ArrayData"), becau
zhengruifeng opened a new pull request, #45809:
URL: https://github.com/apache/spark/pull/45809
### What changes were proposed in this pull request?
Add more tests
### Why are the changes needed?
for test coverage, to address
https://github.com/apache/spark/pull/45788#discussion
leixm commented on PR #45765:
URL: https://github.com/apache/spark/pull/45765#issuecomment-2031015857
> I haven't looked at the code in detail yet, but I have two questions first:
>
> 1. After this PR, which test cases still cover the non-codegen code
branches? The test cases related
panbingkun commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547095372
##
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##
@@ -57,12 +58,14 @@ trait LoggingSuiteBase
def msgWithMDCAndException: Log
panbingkun commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547090080
##
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##
@@ -57,12 +58,14 @@ trait LoggingSuiteBase
def msgWithMDCAndException: Log
maheshk114 commented on PR #45228:
URL: https://github.com/apache/spark/pull/45228#issuecomment-2031012182
> > > I am also concerned about the performance.
> > > I think the best would be if the migration of shuffle data to external
storage would only kick in when the scale down is aggres
panbingkun commented on PR #45808:
URL: https://github.com/apache/spark/pull/45808#issuecomment-2031006783
cc @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
panbingkun commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547090080
##
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##
@@ -57,12 +58,14 @@ trait LoggingSuiteBase
def msgWithMDCAndException: Log
panbingkun commented on code in PR #45808:
URL: https://github.com/apache/spark/pull/45808#discussion_r1547089486
##
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:
##
@@ -29,14 +29,15 @@ import org.apache.logging.log4j.core.filter.AbstractFilter
import org
panbingkun opened a new pull request, #45808:
URL: https://github.com/apache/spark/pull/45808
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
chenhao-db opened a new pull request, #45807:
URL: https://github.com/apache/spark/pull/45807
### What changes were proposed in this pull request?
This PR allows casting from variant to another type. It has the same
semantics as `variant_get` with an empty path.
### How was thi
attilapiros commented on PR #45801:
URL: https://github.com/apache/spark/pull/45801#issuecomment-2030972406
Hi @dongjoon-hyun,
Thanks! I am fine either way.
By the way should not we need to extend the condition with `|| version ==
"4.0"` here:
https://github.com/apache/spa
dongjoon-hyun commented on PR #45804:
URL: https://github.com/apache/spark/pull/45804#issuecomment-2030968203
Thank you, @yaooqinn .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
chenhao-db opened a new pull request, #45806:
URL: https://github.com/apache/spark/pull/45806
### What changes were proposed in this pull request?
This PR adds a new `VariantSchema` expression. It returns schema in the SQL
format of a variant.
Usage examples:
```
> SE
yaooqinn commented on PR #45789:
URL: https://github.com/apache/spark/pull/45789#issuecomment-2030952935
Thank you @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific com
dongjoon-hyun commented on PR #45804:
URL: https://github.com/apache/spark/pull/45804#issuecomment-2030946673
Thank you, @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specifi
cloud-fan closed pull request #45708: [SPARK-47551][SQL] Add variant_get
expression.
URL: https://github.com/apache/spark/pull/45708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comm
cloud-fan commented on PR #45708:
URL: https://github.com/apache/spark/pull/45708#issuecomment-2030916612
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
dongjoon-hyun opened a new pull request, #45804:
URL: https://github.com/apache/spark/pull/45804
…
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
hiboyang opened a new pull request, #45803:
URL: https://github.com/apache/spark/pull/45803
### What changes were proposed in this pull request?
Check `spark.shuffle.readHostLocalDisk` config to determine whether read
shuffle block from same local host machine.
### Why are
1 - 100 of 308 matches
Mail list logo
