[GitHub] [spark] srowen commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

srowen commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153736049 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] ueshin opened a new pull request, #40612: [SPARK-42969][CONNECT][TESTS] Fix the comparison the result with Arrow optimization enabled/disabled

ueshin opened a new pull request, #40612: URL: https://github.com/apache/spark/pull/40612 ### What changes were proposed in this pull request? Fixes the comparison the result with Arrow optimization enabled/disabled. ### Why are the changes needed? in `test_arrow`, there

[GitHub] [spark] rangadi commented on a diff in pull request #40586: [SPARK-42939][SS][CONNECT] Core streaming Python API for Spark Connect

rangadi commented on code in PR #40586: URL: https://github.com/apache/spark/pull/40586#discussion_r1153629770 ## connector/connect/common/src/main/protobuf/spark/connect/commands.proto: ## @@ -177,3 +179,97 @@ message WriteOperationV2 { // (Optional) A condition for overwrit

[GitHub] [spark] rangadi commented on a diff in pull request #40586: [SPARK-42939][SS][CONNECT] Core streaming Python API for Spark Connect

rangadi commented on code in PR #40586: URL: https://github.com/apache/spark/pull/40586#discussion_r1153813344 ## connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -1969,6 +2014,136 @@ class SparkConnectPlanner(val sess

[GitHub] [spark] gengliangwang commented on pull request #40592: [SPARK-42967][CORE][3.2][3.3][3.4] Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is cancelled

gengliangwang commented on PR #40592: URL: https://github.com/apache/spark/pull/40592#issuecomment-1491064274 Merging to master/3.4/3.3/3.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[GitHub] [spark] gengliangwang closed pull request #40592: [SPARK-42967][CORE][3.2][3.3][3.4] Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is cancelled

gengliangwang closed pull request #40592: [SPARK-42967][CORE][3.2][3.3][3.4] Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is cancelled URL: https://github.com/apache/spark/pull/40592 -- This is an automated message from the Apache Git Service. To respond t

[GitHub] [spark] HyukjinKwon commented on pull request #40612: [SPARK-42969][CONNECT][TESTS] Fix the comparison the result with Arrow optimization enabled/disabled

HyukjinKwon commented on PR #40612: URL: https://github.com/apache/spark/pull/40612#issuecomment-1491101671 Merged to master and branch-3.4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [spark] HyukjinKwon commented on pull request #40595: [SPARK-42970][CONNECT][PYTHON][TESTS][3.4] Reuse pyspark.sql.tests.test_arrow test cases

HyukjinKwon commented on PR #40595: URL: https://github.com/apache/spark/pull/40595#issuecomment-1491102080 Merged to branch-3.4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [spark] HyukjinKwon closed pull request #40612: [SPARK-42969][CONNECT][TESTS] Fix the comparison the result with Arrow optimization enabled/disabled

HyukjinKwon closed pull request #40612: [SPARK-42969][CONNECT][TESTS] Fix the comparison the result with Arrow optimization enabled/disabled URL: https://github.com/apache/spark/pull/40612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon closed pull request #40595: [SPARK-42970][CONNECT][PYTHON][TESTS][3.4] Reuse pyspark.sql.tests.test_arrow test cases

HyukjinKwon closed pull request #40595: [SPARK-42970][CONNECT][PYTHON][TESTS][3.4] Reuse pyspark.sql.tests.test_arrow test cases URL: https://github.com/apache/spark/pull/40595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153889706 ## sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala: ## @@ -1742,6 +1742,8 @@ class DataFrameSuite extends QueryTest Seq(Row(2, 1, 2), Row(1

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40608: [SPARK-35198][CORE][PYTHON][SQL] Add support for calling debugCodegen from Python & Java

HyukjinKwon commented on code in PR #40608: URL: https://github.com/apache/spark/pull/40608#discussion_r1153890797 ## python/pyspark/sql/dataframe.py: ## @@ -706,6 +706,25 @@ def explain( assert self._sc._jvm is not None print(self._sc._jvm.PythonSQLUtils.expl

[GitHub] [spark] HeartSaVioR commented on pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on PR #40561: URL: https://github.com/apache/spark/pull/40561#issuecomment-1491105348 > What is the decision about batch support? I just added support of batch in the latest commit. It needs be more test coverage for batch query support so that's why we have new

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40591: [SPARK-42950][CORE] Add exit code in SparkListenerApplicationEnd

HyukjinKwon commented on code in PR #40591: URL: https://github.com/apache/spark/pull/40591#discussion_r1153892668 ## core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala: ## @@ -289,7 +289,8 @@ case class SparkListenerApplicationStart( driverAttributes: Optio

[GitHub] [spark] itholic commented on pull request #39937: [SPARK-42309][SQL] Introduce `INCOMPATIBLE_DATA_TO_TABLE` and sub classes.

itholic commented on PR #39937: URL: https://github.com/apache/spark/pull/39937#issuecomment-1491110788 Test passed. @MaxGekk could you take a look when you find some time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153895088 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153895088 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153895796 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153896106 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153896257 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153896951 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153897632 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153895796 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] zhengruifeng commented on pull request #40612: [SPARK-42969][CONNECT][TESTS] Fix the comparison the result with Arrow optimization enabled/disabled

zhengruifeng commented on PR #40612: URL: https://github.com/apache/spark/pull/40612#issuecomment-1491119721 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

[GitHub] [spark] WweiL commented on a diff in pull request #40586: [SPARK-42939][SS][CONNECT] Core streaming Python API for Spark Connect

WweiL commented on code in PR #40586: URL: https://github.com/apache/spark/pull/40586#discussion_r1153899173 ## connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala: ## @@ -2120,7 +2130,6 @@ class SparkConnectPlanner(val session:

[GitHub] [spark] LuciferYang commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

LuciferYang commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r115394 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] github-actions[bot] commented on pull request #39130: [SPARK-xxxxx][DOCUMENTATION][PYTHON] Fix grammar in docstring for toDF().

github-actions[bot] commented on PR #39130: URL: https://github.com/apache/spark/pull/39130#issuecomment-1491126263 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

[GitHub] [spark] github-actions[bot] closed pull request #39102: [SPARK-41555][SQL] Multi sparkSession should share single SQLAppStatusStore

github-actions[bot] closed pull request #39102: [SPARK-41555][SQL] Multi sparkSession should share single SQLAppStatusStore URL: https://github.com/apache/spark/pull/39102 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [spark] github-actions[bot] closed pull request #38732: [SPARK-41210][K8S] Window based executor failure tracking mechanism

github-actions[bot] closed pull request #38732: [SPARK-41210][K8S] Window based executor failure tracking mechanism URL: https://github.com/apache/spark/pull/38732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153904044 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153895088 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] rangadi commented on a diff in pull request #40586: [SPARK-42939][SS][CONNECT] Core streaming Python API for Spark Connect

rangadi commented on code in PR #40586: URL: https://github.com/apache/spark/pull/40586#discussion_r1153904387 ## python/pyspark/sql/connect/readwriter.py: ## @@ -37,7 +37,7 @@ from pyspark.sql.connect._typing import ColumnOrName, OptionalPrimitiveType from pyspark.sql

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153906537 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153906537 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] rangadi commented on a diff in pull request #40586: [SPARK-42939][SS][CONNECT] Core streaming Python API for Spark Connect

rangadi commented on code in PR #40586: URL: https://github.com/apache/spark/pull/40586#discussion_r1153908437 ## python/pyspark/sql/connect/session.py: ## @@ -489,10 +495,6 @@ def sparkContext(self) -> Any: def streams(self) -> Any: raise NotImplementedError("stre

[GitHub] [spark] rangadi commented on a diff in pull request #40586: [SPARK-42939][SS][CONNECT] Core streaming Python API for Spark Connect

rangadi commented on code in PR #40586: URL: https://github.com/apache/spark/pull/40586#discussion_r1153908962 ## python/pyspark/sql/connect/session.py: ## @@ -14,6 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License.

[GitHub] [spark] rangadi commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

rangadi commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153914621 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] rangadi commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

rangadi commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153915130 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] rangadi commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

rangadi commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153915713 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] rangadi commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

rangadi commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153915858 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] LuciferYang commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

LuciferYang commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153917326 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] sadikovi commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

sadikovi commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153918589 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] srowen commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

srowen commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153918687 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] srowen commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

srowen commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153918827 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] LuciferYang commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

LuciferYang commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153917326 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] LuciferYang commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

LuciferYang commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153920360 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] LuciferYang commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

LuciferYang commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153925550 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] LuciferYang opened a new pull request, #40613: Revert "[SPARK-39204][CORE] Change `Utils.createTempDir` and `Utils.createDirectory` call the same logic method in `JavaUtils`

LuciferYang opened a new pull request, #40613: URL: https://github.com/apache/spark/pull/40613 This reverts commit 5cb5d1fa66ad9d6e94beb17d3fda3a8f220bc371. ### What changes were proposed in this pull request? ### Why are the changes needed? ### Do

[GitHub] [spark] LuciferYang commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

LuciferYang commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153926757 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] lucaspompeun opened a new pull request, #40614: correction of protobuf sql docuentation

lucaspompeun opened a new pull request, #40614: URL: https://github.com/apache/spark/pull/40614 ### What changes were proposed in this pull request? Correction of code highlights in SQL protobuf documentation. old version: ![image](https://user-images.githubusercont

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: Revert "[SPARK-39204][CORE] Change `Utils.createTempDir` and `Utils.createDirectory` call the same logic method in `JavaUtils`

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153928232 ## core/src/main/scala/org/apache/spark/util/Utils.scala: ## @@ -330,7 +351,9 @@ private[spark] object Utils extends Logging { def createTempDir( root: St

[GitHub] [spark] cloud-fan commented on pull request #40601: [SPARK-42975][SQL] Cast result type to timestamp type for string +/- interval

cloud-fan commented on PR #40601: URL: https://github.com/apache/spark/pull/40601#issuecomment-1491166217 The change makes sense, but I'd say this is a legacy feature and the existing behavior doesn't make sense at all. For string +/- internal, the string can be timestamp, timestamp_ntz and

[GitHub] [spark] cloud-fan commented on pull request #40601: [SPARK-42975][SQL] Cast result type to timestamp type for string +/- interval

cloud-fan commented on PR #40601: URL: https://github.com/apache/spark/pull/40601#issuecomment-1491166746 Or we should probably fail it in ANSI mode, cc @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [spark] lucaspompeun commented on pull request #40614: [SPARK-42987][DOCS] Correction of protobuf sql documentation

lucaspompeun commented on PR #40614: URL: https://github.com/apache/spark/pull/40614#issuecomment-1491167649 I'have corrected the problem that cause build error in github workflow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: Revert "[SPARK-39204][CORE] Change `Utils.createTempDir` and `Utils.createDirectory` call the same logic method in `JavaUtils`

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153930483 ## core/src/main/scala/org/apache/spark/util/Utils.scala: ## @@ -320,7 +320,28 @@ private[spark] object Utils extends Logging { * newly created, and is not mark

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153930902 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153931344 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] LuciferYang commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

LuciferYang commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153925550 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] RyanBerti opened a new pull request, #40615: [SPARK-16484][SQL] Add support for Datasketches HllSketch

RyanBerti opened a new pull request, #40615: URL: https://github.com/apache/spark/pull/40615 ### What changes were proposed in this pull request? This PR adds a new dependency on the datasketches-java project, and provides 3 new functions which utilize Datasketches HllSketch and Union ins

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153932172 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153932172 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] LuciferYang closed pull request #40598: [SPARK-42974][CORE] Restore `Utils#createTempDir` use `ShutdownHookManager#registerShutdownDeleteDir` to cleanup tempDir

LuciferYang closed pull request #40598: [SPARK-42974][CORE] Restore `Utils#createTempDir` use `ShutdownHookManager#registerShutdownDeleteDir` to cleanup tempDir URL: https://github.com/apache/spark/pull/40598 -- This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: [SPARK-42974][CORE] Separate Implementation of `Utils.createTempDir` and `JavaUtils.createTempDir`

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153934642 ## core/src/main/scala/org/apache/spark/util/Utils.scala: ## @@ -320,7 +320,28 @@ private[spark] object Utils extends Logging { * newly created, and is not mark

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153935588 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala: ## @@ -980,3 +1022,65 @@ object StreamingDeduplicateExec { private v

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153935792 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala: ## @@ -980,3 +1022,65 @@ object StreamingDeduplicateExec { private v

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153936004 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala: ## @@ -980,3 +1022,65 @@ object StreamingDeduplicateExec { private v

[GitHub] [spark] LuciferYang commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

LuciferYang commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153926757 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] yaooqinn commented on pull request #40602: [SPARK-42978][SQL] Derby&PG: RENAME cannot qualify a new-table-Name with a schema-Name

yaooqinn commented on PR #40602: URL: https://github.com/apache/spark/pull/40602#issuecomment-1491192780 cc @cloud-fan @HyukjinKwon thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153951716 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala: ## @@ -980,3 +1022,65 @@ object StreamingDeduplicateExec { private v

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: [SPARK-42974][CORE] Restore `Utils.createTempDir` to use the `ShutdownHookManager` and clean up `JavaUtils.createTempDir` meth

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153955764 ## common/network-common/src/test/java/org/apache/spark/network/StreamTestHelper.java: ## @@ -49,7 +49,7 @@ private static ByteBuffer createBuffer(int bufSize) {

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: [SPARK-42974][CORE] Restore `Utils.createTempDir` to use the `ShutdownHookManager` and clean up `JavaUtils.createTempDir` meth

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153956628 ## common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalBlockHandlerSuite.java: ## @@ -125,7 +125,7 @@ private void checkDiagnosisResult(

[GitHub] [spark] hvanhovell commented on a diff in pull request #40610: [SPARK-42626][CONNECT] Add Destructive Iterator for SparkResult

hvanhovell commented on code in PR #40610: URL: https://github.com/apache/spark/pull/40610#discussion_r1153957184 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala: ## @@ -45,7 +45,7 @@ private[sql] class SparkResult[T]( priv

[GitHub] [spark] hvanhovell commented on a diff in pull request #40610: [SPARK-42626][CONNECT] Add Destructive Iterator for SparkResult

hvanhovell commented on code in PR #40610: URL: https://github.com/apache/spark/pull/40610#discussion_r1153957374 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala: ## @@ -134,24 +134,41 @@ private[sql] class SparkResult[T](

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: [SPARK-42974][CORE] Restore `Utils.createTempDir` to use the `ShutdownHookManager` and clean up `JavaUtils.createTempDir` meth

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153957701 ## common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/TestShuffleDataContext.java: ## @@ -47,8 +47,9 @@ public TestShuffleDataContext(int numLoca

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: [SPARK-42974][CORE] Restore `Utils.createTempDir` to use the `ShutdownHookManager` and clean up `JavaUtils.createTempDir` meth

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153957872 ## common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java: ## @@ -243,7 +243,9 @@ protected void serviceInit(Configuration external

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: [SPARK-42974][CORE] Restore `Utils.createTempDir` to use the `ShutdownHookManager` and clean up `JavaUtils.createTempDir` meth

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153958391 ## core/src/test/java/test/org/apache/spark/Java8RDDAPISuite.java: ## @@ -246,7 +246,7 @@ public void mapPartitions() { @Test public void sequenceFile() thr

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: [SPARK-42974][CORE] Restore `Utils.createTempDir` to use the `ShutdownHookManager` and clean up `JavaUtils.createTempDir` meth

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153958604 ## core/src/test/java/test/org/apache/spark/JavaAPISuite.java: ## @@ -93,7 +94,7 @@ public class JavaAPISuite implements Serializable { @Before public void se

[GitHub] [spark] LuciferYang commented on a diff in pull request #40613: [SPARK-42974][CORE] Restore `Utils.createTempDir` to use the `ShutdownHookManager` and clean up `JavaUtils.createTempDir` meth

LuciferYang commented on code in PR #40613: URL: https://github.com/apache/spark/pull/40613#discussion_r1153958746 ## streaming/src/test/java/test/org/apache/spark/streaming/JavaAPISuite.java: ## @@ -1476,7 +1476,7 @@ public void testCheckpointMasterRecovery() throws Interrupte

[GitHub] [spark] LuciferYang commented on a diff in pull request #36529: [SPARK-39102][CORE][SQL][DSTREAM] Add checkstyle rules to disabled use of Guava's `Files.createTempDir()`

LuciferYang commented on code in PR #36529: URL: https://github.com/apache/spark/pull/36529#discussion_r1153926757 ## common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java: ## @@ -362,6 +364,60 @@ public static byte[] bufferToArray(ByteBuffer buffer) {

[GitHub] [spark] gengliangwang commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

gengliangwang commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153964890 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache So

[GitHub] [spark] gengliangwang commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

gengliangwang commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153965249 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala: ## @@ -980,3 +1022,65 @@ object StreamingDeduplicateExec { private

[GitHub] [spark] wangyum commented on pull request #40601: [SPARK-42975][SQL] Cast result type to timestamp type for string +/- interval

wangyum commented on PR #40601: URL: https://github.com/apache/spark/pull/40601#issuecomment-1491229091 +1 for fail it in ANSI mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [spark] gengliangwang commented on pull request #40601: [SPARK-42975][SQL] Cast result type to timestamp type for string +/- interval

gengliangwang commented on PR #40601: URL: https://github.com/apache/spark/pull/40601#issuecomment-1491232046 > My suggestion is don't touch it to keep legacy workloads running. We should update the SQL queries to not use String so extensively. +1, totally agree! -- This is an auto

[GitHub] [spark] Hisoka-X commented on a diff in pull request #40609: [SPARK-42316][SQL] Assign name to _LEGACY_ERROR_TEMP_2044

Hisoka-X commented on code in PR #40609: URL: https://github.com/apache/spark/pull/40609#discussion_r1153973500 ## sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala: ## @@ -625,6 +625,20 @@ class QueryExecutionErrorsSuite } } + test("

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153973985 ## sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingDeduplicationWithinWatermarkSuite.scala: ## @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Soft

[GitHub] [spark] Hisoka-X commented on a diff in pull request #40609: [SPARK-42316][SQL] Assign name to _LEGACY_ERROR_TEMP_2044

Hisoka-X commented on code in PR #40609: URL: https://github.com/apache/spark/pull/40609#discussion_r1153975175 ## sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala: ## @@ -625,6 +625,20 @@ class QueryExecutionErrorsSuite } } + test("

[GitHub] [spark] cloud-fan commented on a diff in pull request #40545: [SPARK-42918] Generalize handling of metadata attributes in FileSourceStrategy

cloud-fan commented on code in PR #40545: URL: https://github.com/apache/spark/pull/40545#discussion_r1153976307 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala: ## @@ -220,9 +220,20 @@ object FileSourceStrategy extends Strategy wit

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #40561: [SPARK-42931][SS] Introduce dropDuplicatesWithinWatermark

HeartSaVioR commented on code in PR #40561: URL: https://github.com/apache/spark/pull/40561#discussion_r1153976547 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala: ## @@ -980,3 +1022,65 @@ object StreamingDeduplicateExec { private v

[GitHub] [spark] cloud-fan commented on a diff in pull request #40602: [SPARK-42978][SQL] Derby&PG: RENAME cannot qualify a new-table-Name with a schema-Name

cloud-fan commented on code in PR #40602: URL: https://github.com/apache/spark/pull/40602#discussion_r1153979914 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala: ## @@ -398,10 +398,24 @@ abstract class JdbcDialect extends Serializable with Logging {

[GitHub] [spark] hvanhovell commented on a diff in pull request #40610: [SPARK-42626][CONNECT] Add Destructive Iterator for SparkResult

hvanhovell commented on code in PR #40610: URL: https://github.com/apache/spark/pull/40610#discussion_r1153980266 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala: ## @@ -134,24 +134,41 @@ private[sql] class SparkResult[T](

[GitHub] [spark] cloud-fan commented on a diff in pull request #40602: [SPARK-42978][SQL] Derby&PG: RENAME cannot qualify a new-table-Name with a schema-Name

cloud-fan commented on code in PR #40602: URL: https://github.com/apache/spark/pull/40602#discussion_r1153980662 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/DB2Dialect.scala: ## @@ -113,8 +114,9 @@ private object DB2Dialect extends JdbcDialect { // scalastyle:off lin

[GitHub] [spark] cloud-fan commented on pull request #32987: [SPARK-35564][SQL] Support subexpression elimination for conditionally evaluated expressions

cloud-fan commented on PR #32987: URL: https://github.com/apache/spark/pull/32987#issuecomment-1491247967 After taking another thought, I think the idea is valid. If a subexpression will be evaluated at least once, and likely more than once due to conditional branches, it should be benefici

[GitHub] [spark] hvanhovell commented on a diff in pull request #40611: [SPARK-42981][CONNECT] Add direct arrow serialization

hvanhovell commented on code in PR #40611: URL: https://github.com/apache/spark/pull/40611#discussion_r1153984573 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala: ## @@ -0,0 +1,529 @@ +/* + * Licensed to the Apache S

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40589: [SPARK-38697][SQL] Extend SparkSessionExtensions to inject rules into AQE query stage optimizer

dongjoon-hyun commented on code in PR #40589: URL: https://github.com/apache/spark/pull/40589#discussion_r1153984929 ## sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala: ## @@ -500,6 +500,22 @@ class SparkSessionExtensionSuite extends SparkFunSuite

[GitHub] [spark] yaooqinn commented on pull request #40583: [SPARK-42955][SQL] Skip classifyException and wrap AnalysisException for SparkThrowable

yaooqinn commented on PR #40583: URL: https://github.com/apache/spark/pull/40583#issuecomment-1491249761 cc @cloud-fan @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40589: [SPARK-38697][SQL] Extend SparkSessionExtensions to inject rules into AQE query stage optimizer

dongjoon-hyun commented on code in PR #40589: URL: https://github.com/apache/spark/pull/40589#discussion_r1153985168 ## sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala: ## @@ -1161,3 +1177,12 @@ object AddLimit extends Rule[LogicalPlan] { case

[GitHub] [spark] LuciferYang commented on pull request #40610: [SPARK-42626][CONNECT] Add Destructive Iterator for SparkResult

LuciferYang commented on PR #40610: URL: https://github.com/apache/spark/pull/40610#issuecomment-1491250312 ``` 2023-03-30T16:09:39.936Z [info] - Dataset result destructive iterator *** FAILED *** (84 milliseconds) 2023-03-30T16:09:39.9382605Z 

[GitHub] [spark] LuciferYang commented on pull request #40605: [SPARK-42958][CONNECT] Refactor `connect-jvm-client-mima-check` to support mima check with avro module

LuciferYang commented on PR #40605: URL: https://github.com/apache/spark/pull/40605#issuecomment-1491252507 GA passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40589: [SPARK-38697][SQL] Extend SparkSessionExtensions to inject rules into AQE query stage optimizer

dongjoon-hyun commented on code in PR #40589: URL: https://github.com/apache/spark/pull/40589#discussion_r1153988178 ## sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala: ## @@ -111,11 +112,12 @@ class SparkSessionExtensions { type FunctionDescription

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #40589: [SPARK-38697][SQL] Extend SparkSessionExtensions to inject rules into AQE query stage optimizer

dongjoon-hyun commented on code in PR #40589: URL: https://github.com/apache/spark/pull/40589#discussion_r1153989800 ## sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala: ## @@ -111,11 +112,12 @@ class SparkSessionExtensions { type FunctionDescription

<    1   2   3   >