[GitHub] [spark] SparkQA commented on pull request #33225: [SPARK-35983][SQL] Allow from_json/to_json for map types where value types are day-time intervals

SparkQA commented on pull request #33225: URL: https://github.com/apache/spark/pull/33225#issuecomment-874483254 **[Test build #140682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140682/testReport)** for PR 33225 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

AmplabJenkins removed a comment on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874482159 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140675/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

AmplabJenkins removed a comment on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874482158 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140673/

[GitHub] [spark] sarutak opened a new pull request #33225: [SPARK-35983][SQL] Allow from_json/to_json for map types where value types are day-time intervals

sarutak opened a new pull request #33225: URL: https://github.com/apache/spark/pull/33225 ### What changes were proposed in this pull request? This PR fixes two issues. One is that `to_json` doesn't support `map` types where value types are `day-time` interval types like: ```

[GitHub] [spark] AmplabJenkins commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

AmplabJenkins commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874482159 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140675/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

AmplabJenkins commented on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874482158 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140673/ -- This

[GitHub] [spark] SparkQA commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

SparkQA commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874477156 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45192/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33224: [SPARK-35978][SQL] Support non-reserved keyword TIMESTAMP_LTZ

SparkQA commented on pull request #33224: URL: https://github.com/apache/spark/pull/33224#issuecomment-874475967 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45191/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874475127 **[Test build #140675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140675/testReport)** for PR 33223 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA removed a comment on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874423076 **[Test build #140675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140675/testReport)** for PR 33223 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

SparkQA removed a comment on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874391662 **[Test build #140673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140673/testReport)** for PR 33183 at commit

[GitHub] [spark] SparkQA commented on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

SparkQA commented on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874470943 **[Test build #140673 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140673/testReport)** for PR 33183 at commit

[GitHub] [spark] sarutak commented on pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types

sarutak commented on pull request #32949: URL: https://github.com/apache/spark/pull/32949#issuecomment-874467563 @cloud-fan This is in progress. SPARK-35983 blocks this to fix the issue pointed out [here](https://github.com/apache/spark/pull/32949#discussion_r659884896) but I'll open

[GitHub] [spark] ulysses-you commented on a change in pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

ulysses-you commented on a change in pull request #33222: URL: https://github.com/apache/spark/pull/33222#discussion_r664241710 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantProjects.scala ## @@ -48,9 +48,8 @@ object

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

AmplabJenkins removed a comment on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874465180 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140672/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

AmplabJenkins removed a comment on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874464717 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45190/

[GitHub] [spark] HeartSaVioR commented on pull request #33187: [WIP][SPARK-35988][SS] The implementation for RocksDBStateStoreProvider

HeartSaVioR commented on pull request #33187: URL: https://github.com/apache/spark/pull/33187#issuecomment-874466167 Finally we only have this one for RocksDB state store provider! Please rebase this so that we can continue. -- This is an automated message from the Apache Git Service.

[GitHub] [spark] cloud-fan commented on pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types

cloud-fan commented on pull request #32949: URL: https://github.com/apache/spark/pull/32949#issuecomment-874465537 @sarutak any progress? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AmplabJenkins commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

AmplabJenkins commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874465180 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140672/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

AmplabJenkins commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874464717 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45190/ --

[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874464693 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45190/ -- This is an automated message from the

[GitHub] [spark] SparkQA removed a comment on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

SparkQA removed a comment on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874388349 **[Test build #140672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140672/testReport)** for PR 32934 at commit

[GitHub] [spark] SparkQA commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

SparkQA commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874464256 **[Test build #140672 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140672/testReport)** for PR 32934 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33224: [SPARK-35978][SQL] Support non-reserved keyword TIMESTAMP_LTZ

SparkQA removed a comment on pull request #33224: URL: https://github.com/apache/spark/pull/33224#issuecomment-874460457 **[Test build #140680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140680/testReport)** for PR 33224 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33224: [SPARK-35978][SQL] Support non-reserved keyword TIMESTAMP_LTZ

AmplabJenkins removed a comment on pull request #33224: URL: https://github.com/apache/spark/pull/33224#issuecomment-874462583 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140680/

[GitHub] [spark] SparkQA commented on pull request #33224: [SPARK-35978][SQL] Support non-reserved keyword TIMESTAMP_LTZ

SparkQA commented on pull request #33224: URL: https://github.com/apache/spark/pull/33224#issuecomment-874462559 **[Test build #140680 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140680/testReport)** for PR 33224 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33224: [SPARK-35978][SQL] Support non-reserved keyword TIMESTAMP_LTZ

AmplabJenkins commented on pull request #33224: URL: https://github.com/apache/spark/pull/33224#issuecomment-874462583 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140680/ -- This

[GitHub] [spark] cloud-fan commented on a change in pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

cloud-fan commented on a change in pull request #32959: URL: https://github.com/apache/spark/pull/32959#discussion_r664236165 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala ## @@ -425,12 +467,12 @@ object DateTimeUtils { *

[GitHub] [spark] SparkQA commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

SparkQA commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874460504 **[Test build #140681 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140681/testReport)** for PR 33222 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

cloud-fan commented on a change in pull request #32959: URL: https://github.com/apache/spark/pull/32959#discussion_r664236124 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala ## @@ -368,12 +410,12 @@ object DateTimeUtils { *

[GitHub] [spark] SparkQA commented on pull request #33224: [SPARK-35978][SQL] Support non-reserved keyword TIMESTAMP_LTZ

SparkQA commented on pull request #33224: URL: https://github.com/apache/spark/pull/33224#issuecomment-874460457 **[Test build #140680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140680/testReport)** for PR 33224 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

cloud-fan commented on a change in pull request #32959: URL: https://github.com/apache/spark/pull/32959#discussion_r664235831 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala ## @@ -224,12 +224,12 @@ object DateTimeUtils { *

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

AmplabJenkins removed a comment on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874459546 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45187/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

AmplabJenkins removed a comment on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874459544 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45186/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

AmplabJenkins removed a comment on pull request #33217: URL: https://github.com/apache/spark/pull/33217#issuecomment-874459542 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

AmplabJenkins commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874459546 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45187/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

AmplabJenkins commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874459544 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45186/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

AmplabJenkins commented on pull request #33217: URL: https://github.com/apache/spark/pull/33217#issuecomment-874459542 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] c21 commented on pull request #32944: [SPARK-35794][SQL] Allow custom plugin for AQE cost evaluator

c21 commented on pull request #32944: URL: https://github.com/apache/spark/pull/32944#issuecomment-874459010 Thank you @cloud-fan and @HyukjinKwon for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874453236 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45190/ -- This is an automated message from the Apache

[GitHub] [spark] cloud-fan commented on a change in pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

cloud-fan commented on a change in pull request #33222: URL: https://github.com/apache/spark/pull/33222#discussion_r664229081 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantProjects.scala ## @@ -48,7 +48,7 @@ object RemoveRedundantProjects

[GitHub] [spark] cloud-fan commented on a change in pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

cloud-fan commented on a change in pull request #33222: URL: https://github.com/apache/spark/pull/33222#discussion_r664228339 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantProjects.scala ## @@ -48,7 +48,7 @@ object RemoveRedundantProjects

[GitHub] [spark] cloud-fan commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

cloud-fan commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874451346 The root cause is the wrong mapping between physical and logical plans, and in 3.1 it causes query failures. My fix is to be safe and do not remove `ProjectExec` that may

[GitHub] [spark] SparkQA commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

SparkQA commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874450443 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45187/ -- This is an automated message from the

[GitHub] [spark] gengliangwang opened a new pull request #33224: [SPARK-35978][SQL] Support non-reserved keyword TIMESTAMP_LTZ

gengliangwang opened a new pull request #33224: URL: https://github.com/apache/spark/pull/33224 ### What changes were proposed in this pull request? Support new keyword `TIMESTAMP_LTZ`, which can be used for: - timestamp with local time zone data type in DDL -

[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874448858 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45186/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

SparkQA commented on pull request #33217: URL: https://github.com/apache/spark/pull/33217#issuecomment-874448362 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45188/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

SparkQA commented on pull request #33217: URL: https://github.com/apache/spark/pull/33217#issuecomment-874448237 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45189/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874440343 **[Test build #140679 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140679/testReport)** for PR 32959 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

AmplabJenkins removed a comment on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874439494 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45185/

[GitHub] [spark] AmplabJenkins commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

AmplabJenkins commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874439494 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45185/ --

[GitHub] [spark] SparkQA commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

SparkQA commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874439081 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45187/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874437839 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45186/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

SparkQA commented on pull request #33217: URL: https://github.com/apache/spark/pull/33217#issuecomment-874437696 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45188/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

SparkQA commented on pull request #33217: URL: https://github.com/apache/spark/pull/33217#issuecomment-874437634 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45189/ -- This is an automated message from the Apache

[GitHub] [spark] ulysses-you commented on a change in pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

ulysses-you commented on a change in pull request #33222: URL: https://github.com/apache/spark/pull/33222#discussion_r664216184 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantProjects.scala ## @@ -48,7 +48,7 @@ object

[GitHub] [spark] jerqi commented on a change in pull request #33204: [SPARK-36011][SQL] Disallow altering permanent views based on temporary views or UDFs

jerqi commented on a change in pull request #33204: URL: https://github.com/apache/spark/pull/33204#discussion_r664211365 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ## @@ -259,7 +259,10 @@ case class AlterViewAsCommand( def

[GitHub] [spark] AngersZhuuuu commented on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

AngersZh commented on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874429011 > > Our prod have a very complex job and spark version is 3.1.2. I have tried to put it in UT but it run as normal. But this error actually happen in our prod. With this

[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874428959 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45185/ -- This is an automated message from the

[GitHub] [spark] linhongliu-db commented on a change in pull request #33204: [SPARK-36011][SQL] Disallow altering permanent views based on temporary views or UDFs

linhongliu-db commented on a change in pull request #33204: URL: https://github.com/apache/spark/pull/33204#discussion_r664207369 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ## @@ -259,7 +259,10 @@ case class AlterViewAsCommand(

[GitHub] [spark] cloud-fan commented on a change in pull request #33204: [SPARK-36011][SQL] Disallow altering permanent views based on temporary views or UDFs

cloud-fan commented on a change in pull request #33204: URL: https://github.com/apache/spark/pull/33204#discussion_r664207077 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala ## @@ -465,4 +465,35 @@ class PersistedViewTestSuite

[GitHub] [spark] SparkQA commented on pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

SparkQA commented on pull request #33217: URL: https://github.com/apache/spark/pull/33217#issuecomment-874425604 **[Test build #140678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140678/testReport)** for PR 33217 at commit

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

AngersZh commented on a change in pull request #33217: URL: https://github.com/apache/spark/pull/33217#discussion_r664205457 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala ## @@ -119,6 +119,39 @@ object IntervalUtils {

[GitHub] [spark] viirya commented on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

viirya commented on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874424250 > Our prod have a very complex job and spark version is 3.1.2. I have tried to put it in UT but it run as normal. But this error actually happen in our prod. With this patch our

[GitHub] [spark] SparkQA commented on pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

SparkQA commented on pull request #33217: URL: https://github.com/apache/spark/pull/33217#issuecomment-874423094 **[Test build #140677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140677/testReport)** for PR 33217 at commit

[GitHub] [spark] SparkQA commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

SparkQA commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874423071 **[Test build #140676 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140676/testReport)** for PR 33222 at commit

[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874423076 **[Test build #140675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140675/testReport)** for PR 33223 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

AmplabJenkins removed a comment on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874330285 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45181/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

AmplabJenkins removed a comment on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874422408 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140674/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

AmplabJenkins removed a comment on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874422407 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140671/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

AmplabJenkins removed a comment on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874422409 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45184/

[GitHub] [spark] AmplabJenkins commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

AmplabJenkins commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874422408 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140674/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

AmplabJenkins commented on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874422407 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140671/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

AmplabJenkins commented on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874422409 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45184/ --

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

AngersZh commented on a change in pull request #33217: URL: https://github.com/apache/spark/pull/33217#discussion_r664202409 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala ## @@ -119,6 +119,39 @@ object IntervalUtils {

[GitHub] [spark] sunpe commented on pull request #33154: [SPARK-35949][CORE]Add `keep-spark-context-alive` arg for to prevent closing spark context after invoking main for some case

sunpe commented on pull request #33154: URL: https://github.com/apache/spark/pull/33154#issuecomment-874422054 Hello @kotlovs . I added a arg called `keep-spark-context-alive` to set whether should keep spark context alive after invoke main method. -- This is an automated message from

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33217: [SPARK-36023][SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

AngersZh commented on a change in pull request #33217: URL: https://github.com/apache/spark/pull/33217#discussion_r664201809 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala ## @@ -119,6 +119,39 @@ object IntervalUtils {

[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874418863 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45185/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA removed a comment on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

SparkQA removed a comment on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874377248 **[Test build #140671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140671/testReport)** for PR 33220 at commit

[GitHub] [spark] SparkQA commented on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

SparkQA commented on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874417730 **[Test build #140671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140671/testReport)** for PR 33220 at commit

[GitHub] [spark] HeartSaVioR commented on pull request #33187: [WIP][SPARK-35988][SS] The implementation for RocksDBStateStoreProvider

HeartSaVioR commented on pull request #33187: URL: https://github.com/apache/spark/pull/33187#issuecomment-874415015 NOTE: This might have some post-review comments from #32934 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HeartSaVioR commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

HeartSaVioR commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874414718 Thanks @xuanyuanking for the contribution! I merged this to master/branch-3.2. Please rebase the next PR. -- This is an automated message from the Apache Git

[GitHub] [spark] SparkQA commented on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

SparkQA commented on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874414208 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45184/ -- This is an automated message from the

[GitHub] [spark] SparkQA removed a comment on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA removed a comment on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874405151 **[Test build #140674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140674/testReport)** for PR 33223 at commit

[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874413838 **[Test build #140674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140674/testReport)** for PR 33223 at commit

[GitHub] [spark] jerqi commented on a change in pull request #33204: [SPARK-36011][SQL] Disallow altering permanent views based on temporary views or UDFs

jerqi commented on a change in pull request #33204: URL: https://github.com/apache/spark/pull/33204#discussion_r664192279 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ## @@ -910,4 +910,20 @@ abstract class SQLViewSuite extends

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

AmplabJenkins removed a comment on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874410589 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45183/

[GitHub] [spark] SparkQA commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

SparkQA commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874410569 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45183/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

AmplabJenkins commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874410589 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45183/ --

[GitHub] [spark] HeartSaVioR closed pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

HeartSaVioR closed pull request #32934: URL: https://github.com/apache/spark/pull/32934 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] viirya commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

viirya commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874409343 Okay for me. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HeartSaVioR commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

HeartSaVioR commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874409174 Thanks! Merging to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] HeartSaVioR edited a comment on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

HeartSaVioR edited a comment on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874409174 Thanks! Merging to master/3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] HeartSaVioR commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

HeartSaVioR commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874408509 Thanks for the update. Let's just merge this in and add up the reviews in the next PR then. It should be pretty much simpler as we will have only one PR for RocksDB state

[GitHub] [spark] viirya commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

viirya commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874406115 I'm in vacation, but I will find some time looking at this. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874405151 **[Test build #140674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140674/testReport)** for PR 33223 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

AmplabJenkins removed a comment on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874404556 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45182/

[GitHub] [spark] AmplabJenkins commented on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

AmplabJenkins commented on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874404556 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45182/ --

[GitHub] [spark] SparkQA commented on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

SparkQA commented on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874403808 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45184/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon opened a new pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

HyukjinKwon opened a new pull request #33223: URL: https://github.com/apache/spark/pull/33223 ### What changes were proposed in this pull request? This PR proposes to bump up the mypy version to 0.910 which is the latest. ### Why are the changes needed? To catch the

  1   2   3   4   5   6   >